Mind’s Eye: Grounded Language Model Reasoning through Simulation
Ruibo Liu1,2, Jason Wei1, Shixiang Shane Gu1, Te-Yen Wu2, Soroush Vosoughi2 Claire Cui1, Denny Zhou1, Andrew M. Dai1 1Google Research, Brain Team, 2Dartmouth College
Abstract
Successful and effective communication between humans and AI relies on a shared experience of the world. By training solely on written text, current language models (LMs) miss the grounded experience of humans in the real-world—their failure to relate language to the physical world causes knowledge to be misrepresented and obvious mistakes in their reasoning. We present Mind’s Eye, a paradigm to ground language model reasoning in the physical world. Given a physical reasoning question, we use a computational physics engine (DeepMind’s MuJoCo) to simulate the possible outcomes, and then use the simulation results as part of the input, which enables language models to perform reasoning. Experiments on 39 tasks in a physics alignment benchmark demonstrate that Mind’s Eye can improve reasoning ability by a large margin (27.9% zero-shot, and 46.0% few-shot absolute accuracy improvement on average). Smaller language models armed with Mind’s Eye can obtain similar performance to models that are 100 $\times$ larger. Finally, we confirm the robustness of Mind’s Eye through ablation studies.
中文速览
语言模型（language model, LM）虽然能背诵大量文字知识，却从未真正"体验"过物理世界，导致它在回答自由落体、碰撞等基础物理问题时频繁出错。为此，研究者提出了 Mind's Eye 框架：先用一个文本转代码模型把物理问题翻译成仿真脚本，再交给物理引擎 MuJoCo 跑模拟，最后把仿真结果作为提示注入语言模型，让它"看到实验结果"再作推理。在专门构建的39个物理对齐子任务基准 Utopia 上，Mind's Eye 平均将零样本准确率提升了27.9个百分点、少样本提升了46.0个百分点，甚至让参数量仅为大模型百分之一的小模型达到了与原版大模型相当的水平。这项工作表明，把领域专用的外部仿真器接入推理流程，是弥补语言模型物理常识缺陷、实现更可靠具身推理的一条高效且可扩展的路径。
原文 arXiv:2210.05359;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2210.05359v1