Mind the Gap: Assessing Temporal Generalization in Neural Language Models
Angeliki Lazaridou ♡△♠ Adhiguna Kuncoro⋆♡△ Elena Gribovskaya⋆♡△ Devang Agrawal♢♡ Adam Liška♢♡ Tayfun Terzi♢ Mai Gimenez♢ Cyprien de Masson d’Autume♢ Tomas Kocisky♡ Sebastian Ruder♡ Dani Yogatama♣ Kris Cao♣ Susannah Young♣ Phil Blunsom♣♠ DeepMind, London, UK Equal contribution. ♠ Project initiation. △ Paper writing. ♢ Project technical infrastructure. ♡ Model design and experiments. ♣ Project support and advice.
Abstract
Our world is open-ended, non-stationary, and constantly evolving; thus what we talk about and how we talk about it change over time. This inherent dynamic nature of language contrasts with the current static language modelling paradigm, which trains and evaluates models on utterances from overlapping time periods. Despite impressive recent progress, we demonstrate that Transformer-XL language models perform worse in the realistic setup of predicting future utterances from beyond their training period, and that model performance becomes increasingly worse with time. We find that, while increasing model size alone—a key driver behind recent progress—does not solve this problem, having models that continually update their knowledge with new information can indeed mitigate this performance degradation over time. Hence, given the compilation of ever-larger language modelling datasets, combined with the growing list of language-model-based NLP applications that require up-to-date factual knowledge about the world, we argue that now is the right time to rethink the static way in which we currently train and evaluate our language models, and develop adaptive language models that can remain
中文速览
语言随时间不断演变，但当前的语言模型训练和评估都建立在"时间无关"的静态假设上，训练集和测试集往往来自重叠的时间段。研究者以 Transformer-XL 为实验对象，将训练数据和测试数据按时间严格分开，发现模型在预测训练期结束后两年内的文章时困惑度最高上升 16%，且随时间推移性能持续下滑，说明静态训练范式会系统性地高估模型的真实泛化能力。进一步实验表明，单纯扩大模型规模并不能解决这一"时间衰退"问题——一个用更新数据训练的小模型，反而能超过缺乏近期数据的更大模型；而通过动态评估（dynamic evaluation）让模型在推理时持续用新数据更新参数，则能有效缓解性能退化。这项研究揭示了当前大语言模型在真实部署场景下的关键缺陷，并呼吁学界在预训练数据中保留时间戳、采用时间分层评估基准，推动开发能随世界变化持续自我更新的自适应语言模型。
原文 arXiv:2102.01951;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2102.01951v2