Teaching language models to support answers with verified quotes
Jacob Menick Equal contributions DeepMind Department of Computer Science, University College London Maja Trebacz Equal contributions DeepMind Vladimir Mikulik Equal contributions DeepMind John Aslanides DeepMind Francis Song DeepMind Martin Chadwick DeepMind Mia Glaese DeepMind Susannah Young DeepMind Lucy Campbell-Gillingam DeepMind Geoffrey Irving DeepMind Nat McAleese DeepMind
Abstract
Recent large language models often answer factual questions correctly. But users can’t trust any given claim a model makes without fact-checking, because language models can hallucinate convincing nonsense. In this work we use reinforcement learning from human preferences (RLHP) to train “open-book” QA models that generate answers whilst also citing specific evidence for their claims, which aids in the appraisal of correctness. Supporting evidence is drawn from multiple documents found via a search engine, or from a single user-provided document. Our 280 billion parameter model, GopherCite, is able to produce answers with high quality supporting evidence and abstain from answering when unsure. We measure the performance of GopherCite by conducting human evaluation of answers to questions in a subset of the NaturalQuestions and ELI5 datasets. The model’s response is found to be high-quality 80% of the time on this Natural Questions subset, and 67% of the time on the ELI5 subset. Abstaining from the third of questions for which it is most unsure improves performance to 90% and 80% respectively, approaching human baselines. However, analysis on the adversarial TruthfulQA dataset shows
中文速览
大语言模型回答问题时经常"一本正经地胡说"，用户无法判断哪些答案可以信任。为此，研究者训练了一个名为 GopherCite 的2800亿参数模型，让它在给出答案的同时，从搜索引擎检索到的网页中直接引用逐字原文作为佐证，并通过人类偏好强化学习（reinforcement learning from human preferences, RLHP）优化引用质量。在人工评测中，该模型在自然问答子集上有80%的回答既合理又有据可查，在"像对五岁小孩解释"（ELI5）子集上达到67%；若允许模型对约三分之一最没把握的问题选择"不作答"，两项指标分别提升至90%和80%，接近人类水准。不过，在对抗性数据集 TruthfulQA 上的测试揭示了一个根本局限：能找到"支持证据"并不等于答案就是真的，模型仍可能挑选出误导性引文，说明引用机制只是提升可信度的一环，而非万全之策。
原文 arXiv:2203.11147;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2203.11147v1