Locally Typical Sampling
Clara Meister1 Tiago Pimentel2 Gian Wiher1 Ryan Cotterell1,2 1ETH Zürich 2University of Cambridge
Abstract
Today’s probabilistic language generators fall short when it comes to producing coherent and fluent text despite the fact that the underlying models perform well under standard metrics, e.g., perplexity. This discrepancy has puzzled the language generation community for the last few years. In this work, we posit that the abstraction of natural language generation as a discrete stochastic process—which allows for an information-theoretic analysis—can provide new insights into the behavior of probabilistic language generators, e.g., why high-probability texts can be dull or repetitive. Humans use language as a means of communicating information, aiming to do so in a simultaneously efficient and error-minimizing manner; in fact, psycholinguistics research suggests humans choose each word in a string with this subconscious goal in mind. We formally define the set of strings that meet this criterion: those for which each word has an information content close to the expected information content, i.e., the conditional entropy of our model. We then propose a simple and efficient procedure for enforcing this criterion when generating from probabilistic models, which we call locally typical
中文速览
大型语言模型在困惑度等标准指标上表现优异，却经常生成枯燥、重复、甚至语无伦次的文本，这一矛盾困扰了研究界多年。作者从信息论角度重新审视这个问题，借鉴心理语言学中"均匀信息密度"假说——人类说话时会潜意识地让每个词携带的信息量接近期望值（即条件熵），从而在效率与鲁棒性之间取得平衡。基于这一洞察，作者提出"局部典型采样"（locally typical sampling）：在每一步生成时，只保留信息量与当前上下文条件熵相近的词，剔除信息量过高或过低的候选词，再从这个筛选后的集合中采样。在摘要生成和故事生成任务上的自动与人工评估均表明，与 nucleus 采样和 top-k 采样相比，局部典型采样在保持文本质量的同时显著减少了退化重复，使生成文本的重复率接近真实人类写作水平。这项工作不仅为解释"高概率文本为何质量低下"提供了理论框架，也提供了一个简单高效、已集成进 Hugging Face Transformers 库的实用算法。
原文 arXiv:2202.00666;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2202.00666v6