Teaching Machines to Describe Images via Natural Language Feedback
Huan Ling1, Sanja Fidler1,2 University of Toronto1, Vector Institute2
Abstract
Robots will eventually be part of every household. It is thus critical to enable algorithms to learn from and be guided by non-expert users. In this paper, we bring a human in the loop, and enable a human teacher to give feedback to a learning agent in the form of natural language. We argue that a descriptive sentence can provide a much stronger learning signal than a numeric reward in that it can easily point to where the mistakes are and how to correct them. We focus on the problem of image captioning in which the quality of the output can easily be judged by non-experts. We propose a hierarchical phrase-based captioning model trained with policy gradients, and design a feedback network that provides reward to the learner by conditioning on the human-provided feedback. We show that by exploiting descriptive feedback our model learns to perform better than when given independently written human captions.
中文速览
让机器自动给图片配文字说明时，纯数字奖励信号太粗糙，无法告诉模型"错在哪、该怎么改"。研究者提出让普通人用自然语言（例如"这里应该是狗，不是猫"）来引导一个基于强化学习训练的图像描述模型：他们设计了一个层次化短语生成网络作为描述模型，通过众包平台收集人类对模型输出的文字反馈，再用一个"反馈网络"将这些自然语言转化为奖励信号，代入策略梯度算法来优化模型。实验表明，利用自然语言反馈训练的模型，其表现甚至超过了直接用人工标注描述句子训练的基线。这项工作的意义在于，它为非专业用户提供了一种门槛极低、却信息量极高的方式来纠正AI的错误，朝着让普通人也能"教会"家用机器人迈出了重要一步。
原文 arXiv:1706.00130;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1706.00130v2