Reducing conversational agents’ overconfidence through linguistic calibration
Sabrina J. Mielke1,2 Arthur Szlam2 Emily Dinan2 Y-Lan Boureau2 1 Department of Computer Science, Johns Hopkins University 2 Facebook AI Research
Abstract
While improving neural dialogue agents’ factual accuracy is the object of much research, another important aspect of communication, less studied in the setting of neural dialogue, is transparency about ignorance. In this work, we analyze to what extent state-of-the-art chit-chat models are linguistically calibrated in the sense that their verbalized expression of doubt (or confidence) matches the likelihood that the model’s responses are factually incorrect (or correct). We find that these models are poorly calibrated, yet we show that likelihood of correctness can accurately be predicted. By incorporating such metacognitive features into the training of a controllable generation model, we obtain a dialogue agent with greatly improved linguistic calibration.
中文速览
聊天机器人在回答问题时，常常对错误答案表现出十足的自信，这种"不知道自己不知道"的问题至今缺乏系统研究。研究者针对顶尖的闲聊对话模型（chit-chat model），在大规模问答任务上标注了模型回答的事实正确性与语言置信度（linguistic confidence），发现两者严重脱节——模型说话越自信，答案并不越准确，即语言校准（linguistic calibration）极差。但研究同时证明，利用模型的内部表示可以训练出高准确率的"正确性预测器"，并将其作为元认知信号（metacognitive feature）嵌入可控文本生成（controllable generation）框架，从而让模型在给出答案的同时自动调整措辞，在答不准时说"我不确定"，答得准时才肯定表达。这项工作为让对话AI更诚实地暴露自身局限提供了可行路径，对降低用户被错误信息误导的风险具有实际意义。
原文 arXiv:2012.14983;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2012.14983v2