Multilingual Constituency Parsing with Self-Attention and Pre-Training
Nikita Kitaev Steven Cao Dan Klein Computer Science Division University of California, Berkeley
Abstract
We show that constituency parsing benefits from unsupervised pre-training across a variety of languages and a range of pre-training conditions. We first compare the benefits of no pre-training, fastText (Bojanowski et al., 2017; Mikolov et al., 2018), ELMo (Peters et al., 2018), and BERT (Devlin et al., 2018a) for English and find that BERT outperforms ELMo, in large part due to increased model capacity, whereas ELMo in turn outperforms the non-contextual fastText embeddings. We also find that pre-training is beneficial across all 11 languages tested; however, large model sizes (more than 100 million parameters) make it computationally expensive to train separate models for each language. To address this shortcoming, we show that joint multilingual pre-training and fine-tuning allows sharing all but a small number of parameters between ten languages in the final model. The 10x reduction in model size compared to fine-tuning one model per language causes only a 3.2% relative error increase in aggregate. We further explore the idea of joint fine-tuning and show that it gives low-resource languages a way to benefit from the larger datasets of other languages. Finally, we demonstrate n
中文速览
短短几年间，利用大规模无标注文本做预训练来提升成分句法分析（constituency parsing）的效果引发广泛关注，但不同预训练方法的优劣以及多语言场景下的可扩展性问题尚不清晰。研究者系统比较了fastText、ELMo和BERT三种预训练方式对英语句法分析的影响，发现BERT凭借更大的模型容量显著优于ELMo，而ELMo又优于非上下文的fastText；这一规律在测试的全部11种语言上均成立。面对为每种语言单独微调一个BERT模型带来的参数爆炸问题，他们提出将10种语言的BERT参数完全共享、仅为每种语言保留一个轻量级MLP分类头的联合微调方案，在模型缩小10倍的同时平均F1仅下降0.28，而数据稀缺的语言甚至因此受益于其他语言的大规模标注数据。最终，该方法在包括英语（95.8 F1）和中文（91.8 F1）在内的11种语言上全面刷新了最优结果，为多语言句法分析提供了兼顾精度与计算效率的实用方案。
原文 arXiv:1812.11760;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1812.11760v2