Language Models are Multilingual Chain-of-Thought Reasoners
Freda Shi1,2, Mirac Suzgun1,3,∗ Markus Freitag1 Xuezhi Wang1、Suraj Srivats4 Soroush Vosoughi4 Hyung Won Chung1 Yi Tay1、Sebastian Ruder1 Denny Zhou1 Dipanjan Das1 Jason Wei1 1Google Research 2Toyota Technological Institute at Chicago 3Stanford University 4Dartmouth College Equal contribution. Work done during internship at Google Research.
Abstract
We evaluate the reasoning abilities of large language models in multilingual settings. We introduce the Multilingual Grade School Math (MGSM) benchmark, by manually translating 250 grade-school math problems from the GSM8K dataset (Cobbe et al., 2021) into ten typologically diverse languages. We find that the ability to solve MGSM problems via chain-of-thought prompting emerges with increasing model scale, and that models have strikingly strong multilingual reasoning abilities, even in underrepresented languages such as Bengali and Swahili. Finally, we show that the multilingual reasoning abilities of language models extend to other tasks such as commonsense reasoning and word-in-context semantic judgment. The MGSM benchmark is publicly available at https://github.com/google-research/url-nlp.
中文速览
大型语言模型（LLM）在英语数学推理上的能力已有充分验证，但它们在其他语言、尤其是低资源语言上能否同样推理，此前几乎没有系统评估。研究者为此手工将250道小学数学题（GSM8K）翻译成孟加拉语、斯瓦希里语等十种类型各异的语言，构建了多语言小学数学基准MGSM（Multilingual Grade School Math）。实验发现，结合思维链（chain-of-thought）提示的PaLM-540B模型在所有语言上的解题率均超过40%，即便是训练数据中占比不足0.01%的低资源语言也不例外，且用英语写推理步骤的效果与用母语写不相上下，甚至更优。这一能力随模型规模增大而涌现，并可迁移到常识推理等其他多语言任务，说明超大规模语言模型已具备相当程度的跨语言知识迁移能力，英语思维链提示可作为未来多语言推理研究的有效基线。
原文 arXiv:2210.03057;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2210.03057v1