Contrastive Search Is What You Need For Neural Text Generation
Yixuan Su Language Technology Lab, University of Cambridge Nigel Collier Language Technology Lab, University of Cambridge
Abstract
Generating text with autoregressive language models (LMs) is of great importance to many natural language processing (NLP) applications. Previous solutions for this task often produce text that contains degenerative expressions (Welleck et al., 2020) or lacks semantic consistency (Basu et al., 2021). Recently, Su et al. (2022b) introduced a new decoding method, contrastive search, based on the isotropic representation space of the language model and obtained new state of the art on various benchmarks. In addition, Su et al. (2022b) argued that the representations of autoregressive LMs (e.g. GPT-2) are intrinsically anisotropic which is also shared by previous studies (Ethayarajh, 2019). Therefore, to ensure the language model follows an isotropic distribution, Su et al. (2022b) proposed a contrastive learning scheme, i.e. SimCTG, which calibrates the language model’s representations through additional training.
中文速览
大多数自回归语言模型（autoregressive language model）其实并不存在各向异性（anisotropy）问题——先前研究把GPT-2-small这一个模型的特殊现象误当成了普遍规律，而本文通过对16种语言、38个规模从1.17亿到300亿参数的模型进行系统测量，发现只有GPT-2-small和GPT-2-medium两个模型表现出各向异性，其余模型的表征空间均符合各向同性（isotropic）分布。基于这一发现，研究者进一步验证了对比搜索（contrastive search）解码方法无需任何额外训练便可直接应用于现成语言模型，在开放式文本生成、文档摘要、代码生成和机器翻译四项任务上均显著优于贪心搜索、束搜索及各类随机采样方法。人工评估结果表明，在所测16种语言中有12种语言上，对比搜索生成的文本质量已与人类写作水平相当。这项工作不仅纠正了领域内的一个广泛误解，还证明了对比搜索是一种开箱即用、跨语言通用的高质量文本生成方案，对无法承担大模型二次训练成本的实际应用场景尤具价值。
原文 arXiv:2210.14140;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2210.14140v3