TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentation
Li Ding University of Rochester Rochester, NY 14627、Chenliang Xu University of Rochester Rochester, NY 14627
Abstract
Action segmentation as a milestone towards building automatic systems to understand untrimmed videos has received considerable attention in the recent years. It is typically being modeled as a sequence labeling problem but contains intrinsic and sufficient differences than text parsing or speech processing. In this paper, we introduce a novel hybrid temporal convolutional and recurrent network (TricorNet), which has an encoder-decoder architecture: the encoder consists of a hierarchy of temporal convolutional kernels that capture the local motion changes of different actions; the decoder is a hierarchy of recurrent neural networks that are able to learn and memorize long-term action dependencies after the encoding stage. Our model is simple but extremely effective in terms of video sequence labeling. The experimental results on three public action segmentation datasets have shown that the proposed model achieves superior performance over the state of the art.
中文速览
未经剪辑的长视频中，每一帧究竟属于哪个动作，这是动作分割（action segmentation）任务要解决的核心难题——现有方法要么只擅长捕捉局部动作细节，要么只擅长建模动作之间的长程依赖，难以兼顾。为此，作者提出了一种编码器-解码器混合网络 TricorNet：编码器由多层一维时序卷积（temporal convolution）构成，专门提取局部动作变化；解码器则由多层双向长短期记忆网络（Bi-LSTM）构成，在编码之后进一步学习和记忆不同动作间的长程依赖关系。在三个公开动作分割数据集上的实验表明，TricorNet 在多项评估指标上均达到或超越当时最优水平，定性分析也证明它能生成更平滑、更符合动作先后逻辑的标注结果。这项工作的意义在于，它为视频理解领域提供了一个简洁而高效的框架，明确指出视频序列与语音、文本数据的本质差异，并通过卷积与循环网络的有机结合加以针对性解决。
原文 arXiv:1705.07818;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1705.07818v1