Improving neural networks by preventing co-adaptation of feature detectors
G. E. Hinton∗, N. Srivastava, A. Krizhevsky, I. Sutskever and R. R. Salakhutdinov Department of Computer Science, University of Toronto, 6 King’s College Rd, Toronto, Ontario M5S 3G4, Canada ∗To whom correspondence should be addressed; E-mail:
Abstract
When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This “overfitting” is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random “dropout” gives big improvements on many benchmark tasks and sets new records for speech and object recognition.
中文速览
大型神经网络在小数据集上训练时极易过拟合，导致在测试集上表现很差。这篇论文提出了"随机丢弃"（Dropout）方法：在每次训练时随机将一半隐藏单元临时从网络中移除，迫使每个神经元独立学习有用的特征，而不是依赖其他特定神经元的配合，从而打破特征之间的"共适应"现象。测试时则使用包含所有神经元、但输出权重减半的"平均网络"，等价于对指数级数量的子网络做模型集成。实验表明，Dropout 在手写数字识别（MNIST）、语音识别（TIMIT）、图像分类（CIFAR-10、ImageNet）和文本分类（Reuters）等多个任务上均刷新了当时的最优记录，例如将 ImageNet 的错误率从 45.7% 降至 42.4%。这一方法实现简单、计算高效，为深度学习提供了一种强大的正则化工具，至今仍是训练神经网络的标准技巧之一。
原文 arXiv:1207.0580;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1207.0580v1