Adapting Large Language Models for Document-Level Machine Translation
Minghao Wu♡♡\heartsuit♡ Thuy-Trang Vu♡♡\heartsuit♡ Lizhen Qu♡♡\heartsuit♡ George Foster♠♠\spadesuit♠ Gholamreza Haffari♡♡\heartsuit♡ ♡♡\heartsuit♡Monash University ♠♠\spadesuit♠Google Research
Abstract
Large language models (LLMs) have significantly advanced various natural language processing (NLP) tasks. Recent research indicates that moderately-sized LLMs often outperform larger ones after task-specific fine-tuning. This study focuses on adapting LLMs for document-level machine translation (DocMT) for specific language pairs. We first investigate the impact of prompt strategies on translation performance and then conduct extensive experiments using two fine-tuning methods, three LLM backbones, and 18 translation tasks across nine language pairs. Our results show that specialized models can sometimes surpass GPT-4 in translation performance but still face issues like off-target translation due to error propagation in decoding. We provide an in-depth analysis of these LLMs tailored for DocMT, examining translation errors, discourse phenomena, strategies for training and inference, the data efficiency of parallel documents, recent test set evaluations, and zero-shot crosslingual transfer. Our findings highlight the strengths and limitations of LLM-based DocMT models and provide a foundation for future research.
中文速览
针对大型语言模型（LLMs）在文档级机器翻译（document-level machine translation, DocMT）中表现不足的问题，研究团队系统地探索了如何通过微调让70亿参数规模的中等体量LLMs胜任这一任务。他们在九个语言对、18个翻译方向上，比较了参数高效微调（PEFT）和全量微调（FFT）两种方法，并结合两阶段训练策略——先在目标语言单语文档上微调，再在平行文档上微调——对Llama2、Bloom、Vicuna三种模型主干进行了大规模实验。结果显示，经过针对性微调的中等体量模型在部分翻译任务上能够超越GPT-4，同时表现出更强的跨句语境感知能力和更少的翻译错误，但仍面临因解码阶段错误传播导致的"偏目标语言输出"顽疾。研究还发现，基础版LLM比指令微调版更适合作为任务专用微调的骨干，PEFT在整体表现上优于FFT但数据效率不如FFT，这些发现为未来构建高质量文档级翻译系统提供了重要的实践参考。
原文 arXiv:2401.06468;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2401.06468v4