Recognizing Fine-Grained and Composite Activities using Hand-Centric Features and Script Data
Marcus Rohrbach Anna Rohrbach Michaela Regneri Sikandar Amin Mykhaylo Andriluka Manfred Pinkal Bernt Schiele
Abstract
Activity recognition has shown impressive progress in recent years. However, the challenges of detecting fine-grained activities and understanding how they are combined into composite activities have been largely overlooked. In this work we approach both tasks and present a dataset which provides detailed annotations to address them. The first challenge is to detect fine-grained activities, which are defined by low inter-class variability and are typically characterized by fine-grained body motions. We explore how human pose and hands can help to approach this challenge by comparing two pose-based and two hand-centric features with state-of-the-art holistic features. To attack the second challenge, recognizing composite activities, we leverage the fact that these activities are compositional and that the essential components of the activities can be obtained from textual descriptions or scripts.
中文速览
细粒度动作（fine-grained activity）和复合动作（composite activity）的识别长期被计算机视觉领域忽视——现有方法在类别差异明显的粗粒度动作上表现出色，却难以区分"切片"与"切丁"这类细微差别，也无法理解"做炒鸡蛋"这类由多个步骤组合而成的复杂行为。为此，研究者构建了一个厨房场景下的大规模视频数据集MPII Cooking 2，并提出以手部区域为核心的视觉特征提取方法来应对细粒度识别难题，同时引入属性（attribute）分解机制，将复合动作拆解为可跨类别共享的细粒度动作和参与对象，再结合从文本脚本中自动挖掘的属性关联，解决训练数据不足的瓶颈。实验表明，以手为中心的特征在细粒度动作分类和检测上优于传统整体特征，而基于属性的分解策略不仅大幅提升了复合动作的识别性能，还能在完全没有视频训练样本的情况下识别从未见过的新复合动作。这项工作填补了精细化、长时程日常行为理解的数据与方法空白，对人机交互、老年看护等需要深入理解人类行为的应用场景具有重要意义。
原文 arXiv:1502.06648;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1502.06648v2