Learning to Localize and Align Fine-Grained Actions to Sparse Instructions
Meera Hahn Georgia Institute of Technology Nataniel Ruiz Boston University Jean-Baptiste Alayrac PSL Research University Inria Ivan Laptev Inria James M. Rehg Georgia Institute of Technology
Abstract
Automatically generating textual video descriptions that are time-aligned with the video content is a long-standing goal in computer vision. The task is challenging due to the difficulty of bridging the semantic gap between the visual and natural language domains. This paper addresses the task of automatically generating an alignment between a set of instructions and a first person video demonstrating an activity. The sparse descriptions and ambiguity of written instructions create significant alignment challenges. The key to our approach is the use of egocentric cues to generate a concise set of action proposals, which are then matched to recipe steps using object recognition and computational linguistic techniques. We obtain promising results on both the Extended GTEA Gaze+ dataset and the Bristol Egocentric Object Interactions Dataset.
中文速览
把食谱文字和第一人称（egocentric）烹饪视频自动对齐是个难题，核心困难在于食谱描述极度稀疏（平均仅6词/分钟），而视频却包含海量细节，两者之间几乎没有显式的对应关系。作者提出了一个两阶段流水线：先利用第一人称视频特有的线索——手部姿态、头部运动和眼动注视——通过双向LSTM网络定位出视频中"有动作发生"的片段（即动作提案），再对这些片段做物体识别，并结合无监督语言消歧技术，把识别到的操作对象与食谱步骤中的关键词匹配起来，最终为每个食谱步骤标注出对应的视频时间段。在EGTEA Gaze+和Bristol自我中心物体交互两个数据集上的实验表明，该方法优于现有动作提案方法和视频-文本对齐基线。这项工作的意义在于，它无需旁白或手动标注就能打通海量食谱文本与日益增长的第一人称视频库，为机器人学习操作技能和辅助人类执行任务提供了实用基础。
原文 arXiv:1809.08381;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1809.08381v1