FiLM: Visual Reasoning with a General Conditioning Layer
Ethan Perez1,2, Florian Strub4, Harm de Vries1, Vincent Dumoulin1, Aaron Courville1,3 1MILA, Université de Montréal, 2Rice University, 3CIFAR Fellow, 4Univ. Lille, CNRS, Centrale Lille, Inria, UMR 9189 CRIStAL France
Abstract
We introduce a general-purpose conditioning method for neural networks called FiLM: Feature-wise Linear Modulation. FiLM layers influence neural network computation via a simple, feature-wise affine transformation based on conditioning information. We show that FiLM layers are highly effective for visual reasoning — answering image-related questions which require a multi-step, high-level process — a task which has proven difficult for standard deep learning methods that do not explicitly model reasoning. Specifically, we show on visual reasoning tasks that FiLM layers 1) halve state-of-the-art error for the CLEVR benchmark, 2) modulate features in a coherent manner, 3) are robust to ablations and architectural modifications, and 4) generalize well to challenging, new data from few examples or even zero-shot.
中文速览
用一种叫做"特征线性调制"（FiLM，Feature-wise Linear Modulation）的通用条件化方法，能让神经网络在视觉推理任务上表现得更好——即根据文字问题来回答图像相关的复杂问题。FiLM 的核心做法很简单：让处理问题的循环神经网络（RNN）产生一对缩放和平移参数，直接去调制图像卷积网络（CNN）每一层的特征图，从而让模型的视觉处理过程随着问题内容自适应地改变。在权威视觉推理基准 CLEVR 上，FiLM 将当时最好方法的错误率降低了一半，同时表现出良好的可解释性、鲁棒性，以及在少样本甚至零样本场景下的泛化能力。这项工作说明，无需为推理任务专门设计模块化或关系型架构，一个通用的特征调制机制就足以让模型学会复杂的多步视觉推理，具有很强的实用意义。
原文 arXiv:1709.07871;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1709.07871v2