Multi-Instance Visual-Semantic Embedding
Zhou Ren1 Hailin Jin2 Zhe Lin2 Chen Fang2 Alan Yuille1 1University of California, Los Angeles 2Adobe Research
Abstract
Visual-semantic embedding models have been recently proposed and shown to be effective for image classification and zero-shot learning, by mapping images into a continuous semantic label space. Although several approaches have been proposed for single-label embedding tasks, handling images with multiple labels (which is a more general setting) still remains an open problem, mainly due to the complex underlying corresponding relationship between image and its labels. In this work, we present Multi-Instance visual-semantic Embedding model (MIE) for embedding images associated with either single or multiple labels. Our model discovers and maps semantically-meaningful image subregions to their corresponding labels. And we demonstrate the superiority of our method over the state-of-the-art on two tasks, including multi-label image annotation and zero-shot learning.
中文速览
真实世界的图片往往同时带有多个标签（比如一张照片里同时有"日出""云""鸟"），而现有的视觉-语义嵌入（visual-semantic embedding）方法只擅长处理单标签图片，直接把它们拓展到多标签场景时，会因为不同标签在语义空间中位置差距悬殊，导致整张图的嵌入向量被"拉向四方"、落到一个毫无意义的平均位置。为此，作者提出了多实例视觉-语义嵌入模型（Multi-Instance visual-semantic Embedding，MIE），核心思路是：用区域候选方法自动从图片中裁出若干子区域，然后让每个标签去匹配与它语义最接近的那个子区域，而不是强行用整张图去对应所有标签，同时引入排序优化损失，鼓励正确标签在预测列表中排在最前面。在多标签图片标注任务上，MIE比当时最优方法高出4.5%，在零样本学习（zero-shot learning）任务上也优于基线模型，还能额外输出每个标签对应的图像子区域，具备一定的可解释性。这项工作填补了视觉-语义嵌入在多标签场景下的空白，为大规模、开放词汇的图像理解提供了新思路。
原文 arXiv:1512.06963;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1512.06963v1