Low-Resource Parsing with Crosslingual Contextualized Representations
Phoebe Mulcaire♡ Jungo Kasai♡ Noah A. Smith♡♢ ♡Paul G. Allen School of Computer Science、Engineering, University of Washington, Seattle, WA, USA ♢Allen Institute for Artificial Intelligence, Seattle, WA, USA Equal contribution. Random order.
Abstract
Despite advances in dependency parsing, languages with small treebanks still present challenges. We assess recent approaches to multilingual contextual word representations (cwrs), and compare them for crosslingual transfer from a language with a large treebank to a language with a small or nonexistent treebank, by sharing parameters between languages in the parser itself. We experiment with a diverse selection of languages in both simulated and truly low-resource scenarios, and show that multilingual cwrs greatly facilitate low-resource dependency parsing even without crosslingual supervision such as dictionaries or parallel text. Furthermore, we examine the non-contextual part of the learned language models (which we call a “decontextual probe”) to demonstrate that polyglot language models better encode crosslingual lexical correspondence compared to aligned monolingual language models. This analysis provides further evidence that polyglot training is an effective approach to crosslingual transfer.
中文速览
依存句法分析（dependency parsing）在低资源语言上表现很差，根本原因是这些语言缺乏足够的标注树库。研究者比较了两类多语言上下文词表示（contextual word representations, CWR）方法——把多种语言数据放在一起联合训练单个多语言语言模型（joint training），以及先分别训练各语言模型再用词典对齐（retrofitting）——并把它们应用于跨语言依存句法分析迁移任务，覆盖了欧洲语言、阿拉伯语、日语等多种类型差异较大的语言，既有模拟低资源场景也有真实低资源语言（如哈萨克语、维吾尔语）。实验结果表明，联合训练方法在低资源依存解析上显著优于对齐方法，甚至无需任何词典或平行语料的跨语言监督信号；研究者进一步设计了"去上下文探针"（decontextual probe），通过词语翻译任务证明多语言联合训练的语言模型天然编码了比线性对齐更丰富的跨语言词汇对应关系。这一发现对低资源自然语言处理具有重要意义：只需用多语言原始文本联合训练语言模型，就能为资源匮乏的语言提供高质量的跨语言迁移基础，无需昂贵的双语词典或平行语料。
原文 arXiv:1909.08744;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.08744v1