Deep Learning of Representations: Looking Forward
Yoshua Bengio
Abstract
Deep learning research aims at discovering learning algorithms that discover multiple levels of distributed representations, with higher levels representing more abstract concepts. Although the study of deep learning has already led to impressive theoretical results, learning algorithms and breakthrough experiments, several challenges lie ahead. This paper proposes to examine some of these challenges, centering on the questions of scaling deep learning algorithms to much larger models and datasets, reducing optimization difficulties due to ill-conditioning or local minima, designing more efficient and powerful inference and sampling procedures, and learning to disentangle the factors of variation underlying the observed data. It also proposes a few forward-looking research directions aimed at overcoming these challenges.
中文速览
深度学习的核心目标是让机器自动发现数据中多个层次的抽象表示，从而摆脱对人工特征工程的依赖，但这条路上仍面临重重挑战。这篇文章系统梳理了深度学习在2013年前后的主要进展——包括监督深度网络、卷积网络、受限玻尔兹曼机、自编码器等核心方法——并深入剖析了四类关键难题：如何把模型和数据规模扩展到更大量级、如何克服训练时的病态条件和局部极小值、如何设计更高效的推断与采样方法，以及如何让模型学会分离数据背后相互纠缠的生成因素（disentangling factors of variation）。针对每个挑战，作者还提出了具有前瞻性的研究方向，例如利用课程学习、多任务学习和更好的正则化手段来提升泛化能力。这篇文章的价值在于，它不仅是一份权威的领域综述，更为当时深度学习社区指明了下一步最值得攻克的核心科学问题，对此后数年的研究走向产生了深远影响。
原文 arXiv:1305.0445;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1305.0445v2