Including Uncertainty when Learning from Human Corrections
Dylan P. Losey Rice University、Marcia K. O’Malley Rice University
Abstract
It is difficult for humans to efficiently teach robots how to correctly perform a task. One intuitive solution is for the robot to iteratively learn the human’s preferences from corrections, where the human improves the robot’s current behavior at each iteration. When learning from corrections, we argue that while the robot should estimate the most likely human preferences, it should also know what it does not know, and integrate this uncertainty as it makes decisions. We advance the state-of-the-art by introducing a Kalman filter for learning from corrections: this approach obtains the uncertainty of the estimated human preferences. Next, we demonstrate how the estimate uncertainty can be leveraged for active learning and risk-sensitive deployment. Our results indicate that obtaining and leveraging uncertainty leads to faster learning from human corrections.
中文速览
教机器人按照人类偏好完成任务本来很费劲，一种自然的思路是让机器人通过人类的反复纠错来迭代学习，但现有方法只会估计人类偏好的"最可能值"，完全忽视了这个估计本身有多不确定。这篇论文将逆强化学习（inverse reinforcement learning, IRL）中的纠错学习过程重新建模为一个卡尔曼滤波器（Kalman filter）问题，使机器人在每次更新偏好估计的同时，也能追踪这个估计的不确定度。有了不确定度，机器人既可以主动选择最能消除疑惑的纠错请求（主动学习），也可以在学习结束后对自己最不确定的偏好采取保守策略（风险敏感部署）。实验结果表明，引入并利用不确定度能显著加快机器人从人类纠错中学习的速度，为更安全、更高效的人机交互提供了实用框架。
原文 arXiv:1806.02454;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1806.02454v2