in Proc. of EMNLP 2021 CrossFit : A Few-shot Learning Challenge for Cross-task Generalization in NLP
Qinyuan Ye Bill Yuchen Lin Xiang Ren University of Southern California {qinyuany, yuchen.lin,
Abstract
Humans can learn a new language task efficiently with only few examples, by leveraging their knowledge obtained when learning prior tasks. In this paper, we explore whether and how such cross-task generalization ability can be acquired, and further applied to build better few-shot learners across diverse NLP tasks. We introduce CrossFit , a problem setup for studying cross-task generalization ability, which standardizes seen/unseen task partitions, data access during different learning stages, and the evaluation protocols. To instantiate different seen/unseen task partitions in CrossFit and facilitate in-depth analysis, we present the NLP Few-shot Gym, a repository of 160 diverse few-shot NLP tasks created from open-access NLP datasets and converted to a unified text-to-text format. Our analysis reveals that the few-shot learning ability on unseen tasks can be improved via an upstream learning stage using a set of seen tasks. We also observe that the selection of upstream learning tasks can significantly influence few-shot performance on unseen tasks, asking further analysis on task similarity and transferability.111Our code is at https://github.com/INK-USC/CrossFit.
中文速览
跨任务泛化能力（cross-task generalization）是让模型像人一样"学过多个任务之后能快速上手新任务"，但这一能力在NLP领域长期缺乏系统化的研究框架。作者提出了CrossFit这一标准化的实验设置，要求模型先在一批"已见任务"上进行上游学习，再用极少量样本适配从未见过的新任务，并配套构建了包含160个多样化NLP任务、统一转换为文本到文本格式的"NLP Few-shot Gym"数据集库。实验对多任务学习和三种元学习算法（MAML、一阶MAML、Reptile）进行了系统比较，发现上游多任务学习确实能提升新任务的少样本表现，而且简单的多任务学习往往优于现有元学习方法；此外，上游任务的选择对迁移效果影响显著，盲目增加上游数据量并不总能带来收益。这项工作为研究"如何让预训练语言模型真正具备跨任务泛化能力"提供了可复现的基准和深入分析，对推动通用语言智能的研究具有重要参考价值。
原文 arXiv:2104.08835;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2104.08835v2