LAMOL: LAnguage MOdeling for Lifelong Language Learning
Fan-Keng Sun MIT Cambridge, MA, USA、Cheng-Hao Ho National Taiwan University Taipei, Taiwan、Hung-Yi Lee National Taiwan University Taipei, Taiwan Equal contribution.Work done while at National Taiwan University.
Abstract
Most research on lifelong learning applies to images or games, but not language. We present LAMOL, a simple yet effective method for lifelong language learning (LLL) based on language modeling. LAMOL replays pseudo-samples of previous tasks while requiring no extra memory or model capacity. Specifically, LAMOL is a language model that simultaneously learns to solve the tasks and generate training samples. When the model is trained for a new task, it generates pseudo-samples of previous tasks for training alongside data for the new task. The results show that LAMOL prevents catastrophic forgetting without any sign of intransigence and can perform five very different language tasks sequentially with only one model. Overall, LAMOL outperforms previous methods by a considerable margin and is only 2–3% worse than multitasking, which is usually considered the LLL upper bound. The source code is available at https://github.com/jojotenya/LAMOL.
中文速览
终身语言学习（lifelong language learning）领域长期缺乏能跨越本质不同任务的有效方案，LAMOL 就是为此而生的。它的核心思路是把一个预训练语言模型（language model，GPT-2）同时训练成"解题器"和"出题器"：模型学会回答各类 NLP 问题的同时，也学会凭借一个生成标记重新生成旧任务的训练样本；每当开始学新任务前，就用模型自己生成的"伪样本"来回放旧知识，从而避免灾难性遗忘（catastrophic forgetting），且不需要额外的生成器、不占用额外内存或模型容量。在问答、情感分析、语义解析等五项差异显著的语言任务上，LAMOL 大幅超越正则化和架构类基线方法，与多任务联合训练的理论上界仅差 2–3%。这项工作证明了用单一语言模型就能顺序习得多种截然不同的语言能力，为通用人工智能中的持续学习研究提供了一条简洁而实用的新路径。
原文 arXiv:1909.03329;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.03329v2