PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World
Rowan Zellers♠ Ari Holtzman♠ Matthew Peters♡ Roozbeh Mottaghi♡ Aniruddha Kembhavi♡ Ali Farhadi♠ Yejin Choi♠♡ ♠Paul G. Allen School of Computer Science、Engineering, University of Washington ♡Allen Institute for Artificial Intelligence https://rowanzellers.com/piglet
Abstract
We propose PIGLeT: a model that learns physical commonsense knowledge through interaction, and then uses this knowledge to ground language. We factorize PIGLeT into a physical dynamics model, and a separate language model. Our dynamics model learns not just what objects are but also what they do: glass cups break when thrown, plastic ones don’t. We then use it as the interface to our language model, giving us a unified model of linguistic form and grounded meaning. PIGLeT can read a sentence, simulate neurally what might happen next, and then communicate that result through a literal symbolic representation, or natural language.
中文速览
人类理解语言离不开对物理世界的感知，但现有大语言模型只靠海量文字训练，缺乏真正的"具身"常识。PIGLeT把物理动态建模与语言建模分开处理：先让智能体在3D仿真环境THOR里反复互动，学会"玻璃杯扔了会碎、塑料杯不会"这类物理规律，再用极少量（500条）自然语言标注把这套物理知识迁移到语言理解与生成中。实验表明，给定一句描述动作的英文句子，PIGLeT预测"接下来发生什么"的正确率超过80%，比参数量大100倍的T5-11B还高出10%以上，生成的自然语言描述也被人类评为比同等规模纯语言模型更准确，甚至能零样本泛化到从未读过的新物体。这项工作说明，把交互驱动的物理常识与语言形式解耦并整合，是突破纯文本模型理解瓶颈的一条可行路径。
原文 arXiv:2106.00188;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.00188v2