Finnish Language Modeling with Deep Transformer Models
Abhilash Jain Aku Ruohe Stig-Arne Grönroos Mikko Kurimo Aalto University Aalto University Aalto University Aalto University
Abstract
Transformers have recently taken the centre stage in language modeling after LSTM’s were considered the dominant model architecture for a long time. In this project, we investigate the performance of the Transformer architectures-BERT and Transformer-XL for the language modeling task. We use a sub-word model setting with the Finnish language and compare it to the previous State of the art (SOTA) LSTM model. BERT achieves a pseudo-perplexity score of 14.5, which is a first such measure achieved as far as we know. Transformer-XL improves upon the perplexity score to 73.58 which is 27% better than the LSTM model.
中文速览
长期以来LSTM是语言模型的主流架构，但Transformer的出现正在改变这一局面；本文针对芬兰语这一形态丰富的黏着语，首次将BERT和Transformer-XL两种Transformer架构引入子词级别的语言建模任务，并与现有最优LSTM模型进行对比。研究采用同一份约1.44亿词的芬兰语语料，用Morfessor进行子词切分，分别训练两种模型：对于双向模型BERT，作者提出了一种近似联合概率的"伪困惑度（pseudo-perplexity）"计算方法，最终得分为14.5；对于单向循环注意力模型Transformer-XL，直接计算标准困惑度，得分为73.58，比此前最优LSTM模型低27%。这项工作证明Transformer架构能够有效迁移到小语种的语言建模场景，为芬兰语自动语音识别等下游任务提供了更强的语言模型基础，同时也为双向模型的困惑度评估提供了一种可复用的近似框架。
原文 arXiv:2003.11562;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2003.11562v2