Partially Shuffling the Training Data to Improve Language Models
Ofir Press Paul G. Allen School of Computer Science、Engineering, University of Washington
Abstract
Although SGD requires shuffling the training data between epochs, currently none of the word-level language modeling systems do this. Naively shuffling all sentences in the training data would not permit the model to learn inter-sentence dependencies. Here we present a method that partially shuffles the training data between epochs. This method makes each batch random, while keeping most sentence ordering intact. It achieves new state of the art results on word-level language modeling on both the Penn Treebank and WikiText-2 datasets.111Our code is available at https://github.com/ofirpress/PartialShuffle
中文速览
训练语言模型时，SGD理论上要求每轮训练都对数据随机打乱，但现有词级语言模型（word-level language model）从不这样做，因为直接打乱句子会破坏句间依赖关系，损害模型性能。为此，作者提出了"部分打乱"（Partial Shuffle）方法：将训练序列按批次分成若干行后，对每一行随机选取一个切割点，把切割点前的词挪到行尾，从而在保留大部分词序的前提下让每个epoch的批次内容都有所不同。实验在Penn Treebank和WikiText-2两个标准数据集上测试了当时最先进的几个模型，结果显示这一方法在不增加任何参数或超参数的情况下，以每轮不到百分之一秒的额外开销，让所有模型的困惑度均有提升，并刷新了两个数据集上的最优成绩。这项工作说明一个极其简单的数据处理技巧，就能为语言模型带来稳定收益，提示了现有训练流程中一个长期被忽视的优化空间。
原文 arXiv:1903.04167;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1903.04167v2