Offline RL for Natural Language Generation with Implicit Language Q Learning
Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, Sergey Levine UC Berkeley;
Abstract
Large language models distill broad knowledge from text corpora. However, they can be inconsistent when it comes to completing user specified tasks. This issue can be addressed by finetuning such models via supervised learning on curated datasets, or via reinforcement learning. In this work, we propose a novel offline RL method, implicit language Q-learning (ILQL), designed for use on language models, that combines both the flexible utility maximization framework of RL algorithms with the ability of supervised learning to leverage previously collected data, as well as its simplicity and stability. Our method employs a combination of value conservatism alongside an implicit dataset support constraint in learning value functions, which are then used to guide language model generations towards maximizing user-specified utility functions. In addition to empirically validating ILQL, we present a detailed empirical analysis of situations where offline RL can be useful in natural language generation settings, demonstrating how it can be a more effective utility optimizer than prior approaches for end-to-end dialogue, and how it can effectively optimize high variance reward functions based
中文速览
大型语言模型（LLM）虽然掌握海量知识，却很难稳定地完成用户指定的具体任务，而现有的强化学习（RL）方法要么系统复杂、训练不稳定，要么无法真正利用时序组合性来超越数据集中的平均表现。本文提出了一种名为隐式语言Q学习（Implicit Language Q-Learning，ILQL）的离线强化学习方法，它在语言模型上同时训练Q函数和状态价值函数V，通过期望分位回归实现隐式的数据集支撑约束，并在推理时直接用学到的优势值扰动语言模型的输出概率来引导生成，从而避免了训练时采样或依赖外部语言模型的高昂代价。在对话生成和有毒评论控制等任务上的实验表明，ILQL不仅比监督微调和单步策略提取等基线方法更有效地优化用户指定的奖励函数，还能在训练数据质量较差的情况下通过时序组合性"挖掘"出超越数据上限的最优行为。这一工作为NLP从业者提供了一个兼具RL灵活性与监督学习简洁性的实用工具，对于需要在静态数据上对语言模型进行目标导向对齐的场景具有重要意义。
原文 arXiv:2206.11871;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2206.11871v2