A Closer Look at Memorization in Deep Networks
Devansh Arpit Stanisław Jastrzębski Nicolas Ballas David Krueger Emmanuel Bengio Maxinder S. Kanwal Tegan Maharaj Asja Fischer Aaron Courville Yoshua Bengio Simon Lacoste-Julien
Abstract
We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differences in gradient-based optimization of deep neural networks (DNNs) on noise vs. real data. We also demonstrate that for appropriately tuned explicit regularization (e.g., dropout) we can degrade DNN training performance on noise datasets without compromising generalization on real data. Our analysis suggests that the notions of effective capacity which are dataset independent are unlikely to explain the generalization performance of deep networks when trained with gradient based methods because training data itself plays an important role in determining the degree of memorization.
中文速览
深度神经网络（DNN）明明能记住整个训练集，为什么在真实数据上还能泛化得很好？围绕这个未解之谜，研究者系统对比了DNN在真实数据与随机噪声数据上的学习行为，发现两者存在本质差异：在真实数据上，网络会优先捕捉多个样本共享的简单规律，而不是逐一死记硬背；在噪声数据上，网络才真正依赖蛮力记忆，且训练时间随数据集规模急剧增长。进一步实验表明，适当调节Dropout等显式正则化手段，可以专门阻碍噪声记忆，同时几乎不损害真实数据上的泛化性能。这说明，那种独立于数据集的"有效容量"概念无法解释DNN的泛化之谜——训练数据本身的内在结构，才是决定网络究竟选择学习还是记忆的关键因素。
原文 arXiv:1706.05394;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1706.05394v2