Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Wenhao Zhu1,2, Hongyi Liu3, Qingxiu Dong4, Jingjing Xu2 Shujian Huang1 , Lingpeng Kong5, Jiajun Chen1, Lei Li6 1 National Key Laboratory for Novel Software Technology, Nanjing University 2 Shanghai AI Lab 3 Shanghai Jiao Tong University 4 Peking University 5 The University of Hong Kong 6 Language Technologies Institute, Carnegie Mellon University
Abstract
Large language models (LLMs) have demonstrated remarkable potential in handling multilingual machine translation (MMT). In this paper, we systematically investigate the advantages and challenges of LLMs for MMT by answering two questions: 1) How well do LLMs perform in translating massive languages? 2) Which factors affect LLMs’ performance in translation? We thoroughly evaluate eight popular LLMs, including ChatGPT and GPT-4. Our empirical results show that translation capabilities of LLMs are continually involving. GPT-4 has beat the strong supervised baseline NLLB in 40.91% of translation directions but still faces a large gap towards the commercial translation system like Google Translate, especially on low-resource languages. Through further analysis, we discover that LLMs exhibit new working patterns when used for MMT. First, LLM can acquire translation ability in a resource-efficient way and generate moderate translation even on zero-resource languages. Second, instruction semantics can surprisingly be ignored when given in-context exemplars. Third, cross-lingual exemplars can provide better task guidance for low-resource translation than exemplars in the same language pairs
中文速览
多语言大模型的翻译能力到底有多强、又受哪些因素制约，目前学界还缺乏系统性的评估。研究者对包括 ChatGPT 和 GPT-4 在内的八款主流大语言模型（LLM）展开全面测评，覆盖 102 种语言、606 个翻译方向，并与有监督的多语言翻译系统（NLLB、Google Translate 等）进行对比。结果显示，GPT-4 在约 41% 的翻译方向上已超越有监督基线 NLLB，但在低资源语言上与 Google Translate 仍有明显差距；进一步分析还揭示了三个新规律：LLM 能以极少的训练数据习得翻译能力（甚至对从未见过的语言也能给出中等水平译文）、提示中的指令语义在有示例时会被模型"忽视"、以及跨语言示例有时比同语言示例更能帮助低资源翻译。这项研究为理解 LLM 的多语言翻译机制提供了系统性证据，也指出了当前大模型在语言不平衡和低资源场景下亟待突破的瓶颈。
原文 arXiv:2304.04675;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2304.04675v4