Prompting Large Language Model for Machine Translation: A Case Study
Biao Zhang Barry Haddow Alexandra Birch School of Informatics, University of Edinburgh
Abstract
Research on prompting has shown excellent performance with little or even no supervised training across many tasks. However, prompting for machine translation is still under-explored in the literature. We fill this gap by offering a systematic study on prompting strategies for translation, examining various factors for prompt template and demonstration example selection. We further explore the use of monolingual data and the feasibility of cross-lingual, cross-domain, and sentence-to-document transfer learning in prompting. Extensive experiments with GLM-130B Zeng et al. (2022) as the testbed show that 1) the number and the quality of prompt examples matter, where using suboptimal examples degenerates translation; 2) several features of prompt examples, such as semantic similarity, show significant Spearman correlation with their prompting performance; yet, none of the correlations are strong enough; 3) using pseudo parallel prompt examples constructed from monolingual data via zero-shot prompting could improve translation; and 4) improved performance is achievable by transferring knowledge from prompt examples selected in other settings. We finally provide an analysis on the model
中文速览
大型语言模型（LLM）的提示学习（prompting）在众多NLP任务上表现出色，但如何将其用于机器翻译（machine translation）却缺乏系统研究。研究者以GLM-130B为实验平台，全面考察了提示模板设计、示例选择策略、单语数据利用以及跨语言/跨领域/句到篇章迁移等多个维度的影响。实验发现：示例的数量和质量至关重要，使用质量差的示例会明显拖累翻译效果；语义相似度、语言模型打分、序列长度等特征与提示表现存在显著但偏弱的相关性，无法单独作为可靠的选择依据；利用零样本回译/前向翻译构造的伪平行示例可以改善翻译质量；而来自其他语言对或领域的示例在一定程度上也可迁移使用。这项工作为如何在翻译任务中更有效地使用LLM提供了系统性指导，同时也揭示了提示翻译仍面临的复制、幻觉、实体误译等实际问题，为后续研究指明了方向。
原文 arXiv:2301.07069;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2301.07069v2