Language Models are Few-Shot Learners
Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei OpenAI Equal contributionJohns Hopkins University, OpenAI Author contributions listed at end of paper.
Abstract
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions – something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tas
中文速览
大规模语言模型长期依赖"预训练+微调"的范式，而微调往往需要数千乃至数万条标注样本，这在许多实际场景中代价高昂。OpenAI 训练了拥有 1750 亿参数的自回归语言模型 GPT-3，并系统测试它在不做任何参数更新的情况下，仅凭零样本、单样本或少量示例（few-shot）就能完成各种 NLP 任务的能力——所有"学习"都发生在输入文本的上下文窗口里，而非梯度更新中。实验结果表明，GPT-3 在翻译、问答、填空、算术运算乃至生成新闻文章等任务上表现出色，在部分任务上甚至能与经过专门微调的当前最优模型相媲美，而且模型规模越大、提供的示例越多，性能提升越明显。这项工作的意义在于：它证明了单纯靠扩大模型规模就能让语言模型获得类似人类"举一反三"的泛化能力，同时也诚实地揭示了 GPT-3 在自然语言推理等任务上的短板以及训练数据污染、社会偏见等潜在风险，为后续大模型研究奠定了重要基准。
原文 arXiv:2005.14165;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2005.14165v4