arXiv:2303.17651 · 中英对照阅读
Self-Refine: Iterative Refinement with Self-Feedback
中文速览
大语言模型经常第一稿不够好,但传统改进方法往往需要额外训练数据或人工反馈,这项研究想找一种更省事的办法。作者提出 Self-Refine:让同一个模型先写答案,再指出具体问题并据此修改,反复几轮,整个过程不需要额外训练。测试显示,这种方法在对话、编程、数学等七类任务上都能提升表现,平均比直接生成高约20个百分点,许多改进也更受人类青睐。它说明即使是很强的模型,也能在使用时靠自我检查和修改做得更好,为低成本提升模型输出质量提供了简单通用的思路。
摘要
Like humans, large language models (LLMs) do not always generate the best output on their first try. Motivated by how humans refine their written text, we introduce Self-Refine, an approach for improving initial outputs from LLMs through iterative feedback and refinement. The main idea is to generate an initial output using an LLM; then, the same LLM provides feedback for its output and uses it to refine itself, iteratively. Self-Refine does not require any supervised training data, additional training, or reinforcement learning, and instead uses a single LLM as the generator, refiner and the feedback provider. We evaluate Self-Refine across 7 diverse tasks, ranging from dialog response generation to mathematical reasoning, using state-of-the-art (GPT-3.5 and GPT-4) LLMs. Across all evaluated tasks, outputs generated with Self-Refine are preferred by humans and automatic metrics over those generated with the same LLM using conventional one-step generation, improving by $\sim$ 20% absolute on average in task performance. Our work demonstrates that even state-of-the-art LLMs like GPT-4 can be further improved at test-time using our simple, standalone approach.11 1 Code and data at https://selfrefine.info/.
术语表
- large language model (LLM)
- 大语言模型(LLM)
- Self-Refine
- Self-Refine
- iterative self-refinement
- 迭代式自我改进
- iterative refinement
- 迭代式改进
- feedback
- 反馈
- refinement
- 改进
- initial output
- 初始输出
- candidate output
- 候选输出
- one-step generation
- 单步生成
- test-time
- 测试时
- supervised training data
- 监督训练数据
- reinforcement learning
- 强化学习
- refinement model
- 改进模型
- reward model
- 奖励模型
- external supervision
- 外部监督
- few-shot prompting
- 少样本提示
- in-context learning
- 上下文学习
- task-specific prompt
- 任务特定提示
- actionable feedback
- 可执行反馈
- stopping condition
- 停止条件
- stopping indicator
- 停止指示器
- dialogue response generation
- 对话回复生成
- mathematical reasoning
- 数学推理
- natural language generation
- 自然语言生成
- source-code generation
- 源代码生成
- code optimization
- 代码优化
- GPT-3.5
- GPT-3.5
- GPT-4
- GPT-4
- Codex
- Codex
- text-davinci-003
- text-davinci-003