LETI: Learning to Generate from Textual Interactions
Xingyao Wang Hao Peng Reyhaneh Jabbarvand Heng Ji University of Illinois Urbana-Champaign
Abstract
Fine-tuning pre-trained language models (LMs) is essential for enhancing their capabilities. Existing techniques commonly fine-tune on input-output pairs (e.g., instruction tuning) or with numerical rewards that gauge the output quality (e.g., RLHF). We explore LMs’ potential to learn from textual interactions (LETI) that not only check their correctness with binary labels but also pinpoint and explain errors in their outputs through textual feedback. Our focus is the code generation task, where the model produces code based on natural language instructions. This setting invites a natural and scalable way to acquire textual feedback: the error messages and stack traces from code execution using a Python interpreter. LETI iteratively fine-tunes the model, using the LM objective, on a concatenation of natural language instructions, LM-generated programs, and textual feedback. Prepended to this fine-tuning text, a binary reward token is used to differentiate correct and buggy solutions. LETI requires no ground-truth outputs for training and even outperforms a fine-tuned baseline that does. LETI not only improves the performance of LMs on a code generation dataset MBPP, but also genera
中文速览
预训练语言模型在代码生成等任务上往往只能接受"对/错"这类粗粒度的反馈信号来做微调，而无法从更丰富的错误说明中学习。LETI（Learning from Textual Interactions）提出让模型直接从 Python 解释器返回的报错信息和堆栈追踪（stack trace）中学习：每轮迭代里，模型生成代码、解释器执行并返回具体错误文本，这些文本连同正确/错误标记一起被拼接成训练序列，用语言模型目标对模型进行微调。实验表明，LETI 无需任何人工标注的参考代码，在 MBPP 基准上就超越了使用真实代码的监督微调基线，并能迁移到未见过的 HumanEval 题目；与仅用二元反馈相比，文本反馈让模型用不到一半的梯度更新步数达到相同效果。这项工作说明，让语言模型直接"读懂"环境反馈并从中自我改进，是一条比依赖人工标注更高效、更可扩展的训练路径。
原文 arXiv:2305.10314;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.10314v2