Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs
Maarten Sap♠♢ Ronan Le Bras♠ Daniel Fried♢ Yejin Choi♠♡ ♠Allen Institute for AI, Seattle, WA, USA ♢Language Technologies Institute, Carnegie Mellon University, Pittsburgh, USA ♡Paul G. Allen School of Computer Science, University of Washington, Seattle, WA, USA
Abstract
Social intelligence and Theory of Mind (ToM), i.e., the ability to reason about the different mental states, intents, and reactions of all people involved, allow humans to effectively navigate and understand everyday social interactions. As NLP systems are used in increasingly complex social situations, their ability to grasp social dynamics becomes crucial.
中文速览
大型语言模型（如GPT-3）到底有没有"心理理论"（Theory of Mind，ToM）——即推断他人意图、感受和信念的能力？研究者用两个基准测试来检验这个问题：SocialIQa（考察模型能否理解社交互动中各方的意图与反应）和ToMi（考察模型能否追踪人物对客观世界的不同认知，类似经典的"Sally-Ann错误信念"测试）。实验结果显示，即便是最大规模的GPT-3，在SocialIQa上准确率仅约55%、在ToMi心理状态题上仅约60%，远低于人类表现，而且单纯增大模型规模带来的提升十分有限。研究者从语用学理论出发，指出这一缺陷根源于训练数据缺乏真实互动语境、模型架构本身不以人为中心等结构性问题，并由此提出：仅靠"堆规模"并不够，未来需要引入以人为核心的数据与训练范式，才能让AI真正具备社会智能。
原文 arXiv:2210.13312;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2210.13312v2