Automatic Engineering of Long Prompts
Cho-Jui Hsieh, Si Si, Felix X. Yu, Inderjit S. Dhillon Google Inc. {cjhsieh, sisidaisy, felixyu,
Abstract
Large language models (LLMs) have demonstrated remarkable capabilities in solving complex open-domain tasks, guided by comprehensive instructions and demonstrations provided in the form of prompts. However, these prompts can be lengthy, often comprising hundreds of lines and thousands of tokens, and their design often requires considerable human effort. Recent research has explored automatic prompt engineering for short prompts, typically consisting of one or a few sentences. However, the automatic design of long prompts remains a challenging problem due to its immense search space. In this paper, we investigate the performance of greedy algorithms and genetic algorithms for automatic long prompt engineering. We demonstrate that a simple greedy approach with beam search outperforms other methods in terms of search efficiency. Moreover, we introduce two novel techniques that utilize search history to enhance the effectiveness of LLM-based mutation in our search algorithm. Our results show that the proposed automatic long prompt engineering algorithm achieves an average of 9.2% accuracy gain on eight tasks in Big Bench Hard, highlighting the significance of automating prompt designs
中文速览
长达数千个词符（token）的"长提示词"（long prompt）能大幅提升大语言模型（LLM）在复杂推理任务上的表现，但手工设计这类提示词费时费力，且随模型版本迭代频繁失效。研究者提出了一套自动长提示词优化算法：将长提示词拆分为若干句子，借助LLM对每句话进行语义等价的改写变异，再通过带束搜索（beam search）的贪心算法高效筛选更优版本，并创新性地引入两种利用历史搜索记录来定向引导变异方向的技术，使算法能更有针对性地"往好的方向改"。在Big Bench Hard基准的8项任务上，该方法仅用50次训练集评估便平均取得9.2%的绝对准确率提升，且优于遗传算法等同类方案。这项工作首次系统性地将自动提示词工程拓展到长提示词场景，为充分挖掘LLM潜力、降低提示词设计的人工成本提供了切实可行的路径。
原文 arXiv:2311.10117;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2311.10117v1