Knowledge Matters: Importance of Prior Information for Optimization
\nameÇağlar Gülçehre \addrDépartement d’informatique et de recherche opérationnelle Université de Montréal, Montréal, QC, Canada \AND\nameYoshua Bengio \addrDépartement d’informatique et de recherche opérationnelle Université de Montréal, Montréal, QC, Canada
Abstract
We explore the effect of introducing prior information into the intermediate level of deep supervised neural networks for a learning task on which all the black-box state-of-the-art machine learning algorithms tested have failed to learn. We motivate our work from the hypothesis that there is an optimization obstacle involved in the nature of such tasks, and that humans learn useful intermediate concepts from other individuals via a form of supervision or guidance using a curriculum. The experiments we have conducted provide positive evidence in favor of this hypothesis. In our experiments, a two-tiered MLP architecture is trained on a dataset for which each image input contains three sprites, and the binary target class is 1 if all three have the same shape. Black-box machine learning algorithms only got chance on this task. Standard deep supervised neural networks also failed. However, using a particular structure and guiding the learner by providing intermediate targets in the form of intermediate concepts (the presence of each object) allows to nail the task. Much better than chance but imperfect results are also obtained by exploring architecture and optimization variants, poi
中文速览
深度神经网络在某些"组合任务"上彻底失败——比如判断一张图里三个形状各异的俄罗斯方块拼图是否全部相同——所有现成的黑盒机器学习算法在这类任务上的表现都不比随机猜测强。研究者提出，这种失败源于"有效局部极小值"（effective local minima）问题：当任务需要把两个高度非线性的子任务叠加组合时，梯度下降会陷入难以逃脱的糟糕解，就像孤立的人类学习者很难凭一己之力发现高层抽象概念一样。为了验证"引导学习假说"，他们设计了一个两层MLP架构，在中间层引入监督信号——即先明确告诉网络每个位置有什么形状（中间概念），再让第二层学习最终分类，结果网络轻松解决了问题，而没有中间引导的相同架构则完全失败。这项工作的意义在于，它用机器学习实验为"文化引导学习"理论提供了具体证据：人类之所以能习得高层抽象知识，很可能正是因为其他人通过课程式引导（curriculum learning）提供了关键的中间概念，而非靠个体独自摸索——这对如何设计更聪明的AI训练策略具有重要启示。
原文 arXiv:1301.4083;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1301.4083v6