Do Language Models Know When They’re Hallucinating References?
Ayush Agrawal Microsoft Research \AndMirac Suzgun Stanford University \ANDLester Mackey Microsoft Research \AndAdam Tauman Kalai OpenAI Work done while at Microsoft Research.
Abstract
State-of-the-art language models (LMs) are notoriously susceptible to generating hallucinated information. Such inaccurate outputs not only undermine the reliability of these models but also limit their use and raise serious concerns about misinformation and propaganda. In this work, we focus on hallucinated book and article references and present them as the “model organism” of language model hallucination research, due to their frequent and easy-to-discern nature. We posit that if a language model cites a particular reference in its output, then it should ideally possess sufficient information about its authors and content, among other relevant details. Using this basic insight, we illustrate that one can identify hallucinated references without ever consulting any external resources, by asking a set of direct or indirect queries to the language model about the references. These queries can be considered as “consistency checks.” Our findings highlight that while LMs, including GPT-4, often produce inconsistent author lists for hallucinated references, they also often accurately recall the authors of real references. In this sense, the LM can be said to “know” when it is hallucina
中文速览
语言模型在生成学术引用时会"无中生有"地捏造出根本不存在的书目，这不仅让人难以信任模型输出，还可能在法律、医学等严肃场合引发严重后果。研究者聚焦于"幻觉引用"这一具体问题，提出将其作为研究语言模型幻觉现象的"模式生物"，并设计了一套无需查询外部数据库、仅靠向模型本身提问来判断引用真伪的"一致性检验"方法——既包括直接追问"这篇文章是否真实存在"的直接查询，也包括多次独立追问"这篇文章的作者是谁"并比对答案是否吻合的间接查询。实验发现，GPT-4等模型在面对自己捏造的引用时，往往给出前后矛盾的作者列表，而对真实文献则能稳定地回忆出正确作者，说明模型在某种意义上"知道"自己什么时候在撒谎。这一发现意味着幻觉问题的根源更可能出在生成（解码）环节而非训练数据，为通过改进生成流程来减少幻觉提供了重要方向。
原文 arXiv:2305.18248;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.18248v3