MetaICL: Learning to Learn In Context
Sewon Min1,2 Mike Lewis2 Luke Zettlemoyer1,2 Hannaneh Hajishirzi1,3 1University of Washington 2Meta AI 3Allen Institute for AI
Abstract
We introduce MetaICL (Meta-training for In-Context Learning), a new meta-training framework for few-shot learning where a pretrained language model is tuned to do in-context learning on a large set of training tasks. This meta-training enables the model to more effectively learn a new task in context at test time, by simply conditioning on a few training examples with no parameter updates or task-specific templates. We experiment on a large, diverse collection of tasks consisting of 142 NLP datasets including classification, question answering, natural language inference, paraphrase detection and more, across seven different meta-training/target splits. MetaICL outperforms a range of baselines including in-context learning without meta-training and multi-task learning followed by zero-shot transfer. We find that the gains are particularly significant for target tasks that have domain shifts from the meta-training tasks, and that using a diverse set of the meta-training tasks is key to improvements. We also show that MetaICL approaches (and sometimes beats) the performance of models fully finetuned on the target task, and outperforms much bigger models with nearly 8x parameters. Fin
中文速览
大规模预训练语言模型做"上下文学习"（in-context learning）时，只需在输入中拼几个示例就能处理新任务，但效果不稳定、方差高、还常常依赖人工设计的模板。MetaICL 的思路是：在正式测试之前，先用一大批训练任务对模型做"元训练"——每次训练都模拟真实的上下文学习场景，把多个示例拼成一条序列喂给模型，让模型学会从示例中自动理解任务语义，而不依赖任何模板。实验覆盖 142 个 NLP 数据集、七种元训练与目标任务划分，结果显示 MetaICL 全面超越无元训练的上下文学习和多任务零样本迁移基线，在目标任务与训练任务差异较大时优势尤为突出，性能接近甚至超过在目标任务上全量微调的模型，还能媲美参数量近 8 倍的大模型。这项工作的意义在于，它让小规模语言模型也能稳定、高效地"举一反三"，无需人工模板即可快速适配全新任务，同时与人工指令结合还能进一步提升效果。
原文 arXiv:2110.15943;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.15943v2