arXiv:2508.14460 · 中英对照阅读
DuPO:通过双重偏好优化实现可靠的大语言模型自我验证
DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimization
中文速览
DuPO 想解决的是大模型训练太依赖人工偏好或标准答案,遇到翻译这类没有唯一答案的任务时尤其难获得可靠反馈。它不要求模型把答案完整反推回原题,而是把题目拆成已知和未知部分,让模型根据自己的答案补出未知部分,再用补得是否准确作为奖励信号,并据此优化模型。实验中,DuPO让多语种翻译平均提升2.13个COMET分,在数学推理基准上平均提高6.4个百分点;即使不训练,只用它重排答案也能提升9.3个百分点。它的重要性在于提供了一种不靠人工标注、也不局限于有标准答案任务的自我反馈办法,让大模型能更广泛地改进。
摘要
我们提出双重学习式偏好优化(DuPO),这是一种基于双重学习的偏好优化框架,通过广义对偶生成无标注反馈。DuPO 解决了两项关键局限:基于可验证奖励的强化学习(RLVR)依赖成本高昂的标签,且适用范围仅限于可验证任务;传统双重学习则局限于严格对偶的任务对(例如,翻译与回译)。具体而言,DuPO 将原任务的输入分解为已知部分和未知部分,然后构建其对偶任务,利用原任务的输出和已知信息重构未知部分(例如,逆向推导数学解以恢复隐变量),从而拓宽其在不可逆任务中的适用范围。重构质量作为自监督奖励来优化原任务,并与大语言模型通过单一模型实现两项任务的能力相协同。实证结果表明,DuPO 在多种任务上均取得显著提升:在 756 个方向上,翻译质量平均提高 2.13 个 COMET 分数;在三个具有挑战性的基准测试中,数学推理准确率平均提高 6.4 个百分点;作为推理时重排序器,其性能提高 9.3 个百分点(以增加计算量换取准确率)。这些结果表明,DuPO 为大语言模型优化提供了一种可扩展、通用且无标注的范式。
We present DuPO, a dual learning-based preference optimization framework that generates annotation-free feedback via a generalized duality. DuPO addresses two key limitations: Reinforcement Learning with Verifiable Rewards (RLVR)’s reliance on costly labels and applicability restricted to verifiable tasks, and traditional dual learning’s restriction to strictly dual task pairs (e.g., translation and back-translation). Specifically, DuPO decomposes a primal task’s input into known and unknown components, then constructs its dual task to reconstruct the unknown part using the primal output and known information (e.g., reversing math solutions to recover hidden variables), broadening applicability to non-invertible tasks. The quality of this reconstruction serves as a self-supervised reward to optimize the primal task, synergizing with LLMs’ ability to instantiate both tasks via a single model. Empirically, DuPO achieves substantial gains across diverse tasks: it enhances the average translation quality by 2.13 COMET over 756 directions, boosts the mathematical reasoning accuracy by an average of 6.4 points on three challenge benchmarks, and enhances performance by 9.3 points as an inference-time reranker (trading computation for accuracy). These results position DuPO as a scalable, general, and annotation-free paradigm for LLM optimization.
术语表
- DuPO
- 双重学习式偏好优化
- Dual Learning-based Preference Optimization
- 基于双重学习的偏好优化
- Large Language Models (LLMs)
- 大语言模型
- preference optimization
- 偏好优化
- Reinforcement Learning from Human Feedback (RLHF)
- 基于人类反馈的强化学习
- Reinforcement Learning with Verifiable Rewards (RLVR)
- 基于可验证奖励的强化学习
- Reinforcement Learning from Verifiable Rewards
- 基于可验证奖励的强化学习
- dual learning
- 双重学习
- generalized duality
- 广义对偶
- primal task
- 原任务
- dual task
- 对偶任务
- task duality
- 任务对偶性
- known component
- 已知部分
- unknown component
- 未知部分
- self-supervised feedback
- 自监督反馈
- self-supervised reward
- 自监督奖励
- cycle consistency
- 循环一致性
- annotation-free
- 无标注
- bidirectional competence asymmetry
- 双向能力不对称
- task asymmetry
- 任务不对称性
- capability asymmetry
- 能力不对称性
- LLM-as-a-Judge
- 大语言模型评审
- AI-Feedback/RLAIF
- 人工智能反馈/基于人工智能反馈的强化学习
- Constitutional AI
- 宪法式人工智能
- reward model
- 奖励模型
- policy model
- 策略模型
- PPO
- 近端策略优化
- GRPO
- 群体相对策略优化
- COMET
- COMET机器翻译评价指标
- back-translation
- 回译