Multi-Cue Zero-Shot Learning with Strong Supervision
Zeynep Akata, Mateusz Malinowski, Mario Fritz and Bernt Schiele Max-Planck Institute for Informatics
Abstract
Scaling up visual category recognition to large numbers of classes remains challenging. A promising research direction is zero-shot learning, which does not require any training data to recognize new classes, but rather relies on some form of auxiliary information describing the new classes. Ultimately, this may allow to use textbook knowledge that humans employ to learn about new classes by transferring knowledge from classes they know well. The most successful zero-shot learning approaches currently require a particular type of auxiliary information – namely attribute annotations performed by humans – that is not readily available for most classes. Our goal is to circumvent this bottleneck by substituting such annotations by extracting multiple pieces of information from multiple unstructured text sources readily available on the web. To compensate for the weaker form of auxiliary information, we incorporate stronger supervision in the form of semantic part annotations on the classes from which we transfer knowledge. We achieve our goal by a joint embedding framework that maps multiple text parts as well as multiple semantic parts into a common space. Our results consistently and
中文速览
大规模视觉类别识别面临的核心难题在于：识别全新类别时往往没有任何训练样本，现有零样本学习（zero-shot learning）方法虽然可以借助"辅助信息"来迁移知识，但几乎都依赖昂贵的人工属性标注，难以大规模推广。本文提出用网络上唾手可得的非结构化文本（如维基百科文章）来替代人工属性标注，同时在已知类别上引入语义部件（semantic part）标注作为更强的视觉监督信号，以弥补文本信息噪声大、质量参差不齐带来的性能损失。具体做法是构建一个联合嵌入框架，将多种文本表示（属性、Word2Vec、词袋模型及两种新提出的改进方法）与图像的多个语义部件特征共同映射到同一空间，通过部件级别的对齐目标让文本线索与视觉部件相互匹配。实验表明，该方法在细粒度零样本识别与检索任务上显著超越现有最优方法，且在测试阶段无需提供部件标注，这意味着只需公开可得的文本资料就能达到甚至超越依赖昂贵人工标注的系统性能，为零样本学习的大规模实用化铺平了道路。
原文 arXiv:1603.08754;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1603.08754v1