Few-Shot Text Generation with Natural Language Instructions
Timo Schick Hinrich Schütze Center for Information and Language Processing, LMU Munich, Germany
Abstract
Providing pretrained language models with simple task descriptions in natural language enables them to solve some tasks in a fully unsupervised fashion. Moreover, when combined with regular learning from examples, this idea yields impressive few-shot results for a wide range of text classification tasks. It is also a promising direction to improve data efficiency in generative settings, but there are several challenges to using a combination of task descriptions and example-based learning for text generation. In particular, it is crucial to find task descriptions that are easy to understand for the pretrained model and to ensure that it actually makes good use of them; furthermore, effective measures against overfitting have to be implemented. In this paper, we show how these challenges can be tackled: We introduce genPet, a method for text generation that is based on pattern-exploiting training, a recent approach for combining textual instructions with supervised learning that only works for classification tasks. On several summarization and headline generation datasets, genPet gives consistent improvements over strong baselines in few-shot settings.111Our implementation of genPet
中文速览
给预训练语言模型提供自然语言任务说明（task description）能显著提升少样本场景下的文本分类效果，但如何把同样的思路用于文本生成任务却面临三大难题：怎样让模型真正理解指令、如何选出有效的指令模板、以及如何在训练样本极少时防止过拟合。作者提出了 genPet 方法，将专为分类设计的"模式利用训练"（Pattern-Exploiting Training, PET）扩展到生成式场景，通过在输入中嵌入自然语言指令来引导 Pegasus 等编码器-解码器模型，同时结合多模板融合与正则化手段抑制过拟合。在六个英文摘要生成与标题生成数据集上，仅用 10 到 100 条训练样本，genPet 就能稳定超越强基线，证明了将文字指令与极少量标注样本结合是提升生成任务数据效率的一条切实可行的路径。
原文 arXiv:2012.11926;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2012.11926v2