Shaking the foundations: delusions in sequence models for interaction and control
Pedro A. Ortega Deepmind Safety Analysis Markus Kunesch Deepmind Safety Analysis Grégoire Delétang Deepmind Safety Analysis Tim Genewein Deepmind Safety Analysis Jordi Grau-Moya Deepmind Safety Analysis Joel Veness DeepMind Jonas Buchli DeepMind Jonas Degrave DeepMind Bilal Piot DeepMind Julien Perolat DeepMind Tom Everitt DeepMind Corentin Tallec DeepMind Emilio Parisotto DeepMind Tom Erez DeepMind Yutian Chen DeepMind Scott Reed DeepMind Marcus Hutter DeepMind Nando de Freitas DeepMind Shane Legg DeepMind
Abstract
The recent phenomenal success of language models has reinvigorated machine learning research, and large sequence models such as transformers are being applied to a variety of domains. One important problem class that has remained relatively elusive however is purposeful adaptive behavior. Currently there is a common perception that sequence models “lack the understanding of the cause and effect of their actions” leading them to draw incorrect inferences due to auto-suggestive delusions. In this report we explain where this mismatch originates, and show that it can be resolved by treating actions as causal interventions. Finally, we show that in supervised learning, one can teach a system to condition or intervene on data by training with factual and counterfactual error signals respectively.
中文速览
语言模型本质上是对序列数据建立条件概率分布，用它来预测下一个词或动作本无问题，但当模型被用于"主动行动"时——也就是让模型自己生成动作、再把这些动作当作观测证据来更新信念——就会产生一种"自我妄想"：模型把自己生成的输出误解为关于外部世界的信息，从而得出错误推断（比如"我说了奶酪汉堡，所以菜单上就是奶酪汉堡"）。问题的根源在于，观测数据和模型自身生成的动作在因果结构上地位不同：观测是外部世界施加给模型的，应当做贝叶斯条件化处理；而动作是模型自己发出的干预，必须用因果推断中的"do算子"来处理，否则会通过隐藏的混淆变量引入虚假信息流。作者进一步证明，在监督学习中可以通过同时使用"事实误差信号"和"反事实误差信号"来训练模型区分这两种情况，使其真正习得因果干预能力。这一发现对基础模型、行为克隆、逆强化学习以及目标条件策略等大量前沿方向具有根本性意义，指出了当前序列模型用于自主决策时的核心缺陷及其解决路径。
原文 arXiv:2110.10819;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.10819v1