Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine
Wenxiang Jiao Wenxuan Wang Jen-tse Huang Xing Wang Shuming Shi Zhaopeng Tu Tencent AI Lab Corresponding author.
Abstract
This report provides a preliminary evaluation of ChatGPT for machine translation, including translation prompt, multilingual translation, and translation robustness. We adopt the prompts advised by ChatGPT to trigger its translation ability and find that the candidate prompts generally work well with minor performance differences. By evaluating on a number of benchmark test sets, we find that ChatGPT performs competitively with commercial translation products (e.g., Google Translate) on high-resource European languages but lags behind significantly on low-resource or distant languages. As for the translation robustness, ChatGPT does not perform as well as the commercial systems on biomedical abstracts or Reddit comments but exhibits good results on spoken language. Further, we explore an interesting strategy named pivot prompting for distant languages, which asks ChatGPT to translate the source sentence into a high-resource pivot language before into the target language, improving the translation performance noticeably. With the launch of the GPT-4 engine, the translation performance of ChatGPT is significantly boosted, becoming comparable to commercial translation products, even f
中文速览
ChatGPT作为一款通用大语言模型，究竟能不能胜任机器翻译任务？研究者从翻译提示词设计、多语言翻译能力和翻译鲁棒性三个维度对ChatGPT展开系统评测，并与谷歌翻译、DeepL等商业翻译产品进行对比。结果显示，在德英等高资源欧洲语言对上ChatGPT表现与商业系统相当，但在中文、罗马尼亚语等低资源或跨语系语言对上差距明显，在生物医学摘要和社交媒体噪声文本上也不及商业系统，不过在口语翻译上反而更胜一筹。为此，研究者提出"枢轴提示（pivot prompting）"策略——先让ChatGPT将源语言译成英语，再从英语译入目标语言——能显著改善远距离语言对的翻译质量；而升级到GPT-4引擎后，ChatGPT的翻译能力大幅提升，即便是远距离语言对也能媲美商业系统，幻觉和误译错误也最少，这表明ChatGPT已经具备了实用级别的翻译能力。
原文 arXiv:2301.08745;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2301.08745v4