Aha.
正在载入中英对照阅读…

arXiv:2303.17651 · 中英对照阅读

Self-Refine: Iterative Refinement with Self-Feedback

Aman Madaan、Niket Tandon、Prakhar Gupta、Skyler Hallinan、Luyu Gao、Sarah Wiegreffe、Uri Alon、Nouha Dziri、Shrimai Prabhumoye、Yiming Yang、Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann、Sean Welleck、Amir Yazdanbakhsh、Peter Clark

中文速览

大语言模型经常第一稿不够好,但传统改进方法往往需要额外训练数据或人工反馈,这项研究想找一种更省事的办法。作者提出 Self-Refine:让同一个模型先写答案,再指出具体问题并据此修改,反复几轮,整个过程不需要额外训练。测试显示,这种方法在对话、编程、数学等七类任务上都能提升表现,平均比直接生成高约20个百分点,许多改进也更受人类青睐。它说明即使是很强的模型,也能在使用时靠自我检查和修改做得更好,为低成本提升模型输出质量提供了简单通用的思路。

摘要

Like humans, large language models (LLMs) do not always generate the best output on their first try. Motivated by how humans refine their written text, we introduce Self-Refine, an approach for improving initial outputs from LLMs through iterative feedback and refinement. The main idea is to generate an initial output using an LLM; then, the same LLM provides feedback for its output and uses it to refine itself, iteratively. Self-Refine does not require any supervised training data, additional training, or reinforcement learning, and instead uses a single LLM as the generator, refiner and the feedback provider. We evaluate Self-Refine across 7 diverse tasks, ranging from dialog response generation to mathematical reasoning, using state-of-the-art (GPT-3.5 and GPT-4) LLMs. Across all evaluated tasks, outputs generated with Self-Refine are preferred by humans and automatic metrics over those generated with the same LLM using conventional one-step generation, improving by $\sim$ 20% absolute on average in task performance. Our work demonstrates that even state-of-the-art LLMs like GPT-4 can be further improved at test-time using our simple, standalone approach.11 1 Code and data at https://selfrefine.info/.

术语表

large language model (LLM)
大语言模型(LLM)
Self-Refine
Self-Refine
iterative self-refinement
迭代式自我改进
iterative refinement
迭代式改进
feedback
反馈
refinement
改进
initial output
初始输出
candidate output
候选输出
one-step generation
单步生成
test-time
测试时
supervised training data
监督训练数据
reinforcement learning
强化学习
refinement model
改进模型
reward model
奖励模型
external supervision
外部监督
few-shot prompting
少样本提示
in-context learning
上下文学习
task-specific prompt
任务特定提示
actionable feedback
可执行反馈
stopping condition
停止条件
stopping indicator
停止指示器
dialogue response generation
对话回复生成
mathematical reasoning
数学推理
natural language generation
自然语言生成
source-code generation
源代码生成
code optimization
代码优化
GPT-3.5
GPT-3.5
GPT-4
GPT-4
Codex
Codex
text-davinci-003
text-davinci-003