A Closer Look at Memorization in Deep Networks
Devansh Arpit Affiliation: Montréal Institute for Learning Algorithms, Canada Affiliation: Université de Montréal, Canada Stanisław Jastrzębski Affiliation: Jagiellonian University, Krakow, Poland Nicolas Ballas Affiliation: Montréal Institute for Learning Algorithms, Canada Affiliation: Université de Montréal, Canada David Krueger Affiliation: Montréal Institute for Learning Algorithms, Canada Affiliation: Université de Montréal, Canada Emmanuel Bengio Affiliation: McGill University, Canada Maxinder S. Kanwal Affiliation: University of California, Berkeley, USA Tegan Maharaj Affiliation: Montréal Institute for Learning Algorithms, Canada Affiliation: Polytechnique Montréal, Canada Asja Fischer Affiliation: University of Bonn, Bonn, Germany Aaron Courville Affiliation: Montréal Institute for Learning Algorithms, Canada Affiliation: Université de Montréal, Canada Affiliation: CIFAR Fellow Yoshua Bengio Affiliation: Montréal Institute for Learning Algorithms, Canada Affiliation: Université de Montréal, Canada Affiliation: CIFAR Senior Fellow Simon Lacoste-Julien Correspondence to: Affiliation: Montréal Institute for Learning Algorithms, Canada Affiliation: Université de Montréal, Canada
Abstract
We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differences in gradient-based optimization of deep neural networks (DNNs) on noise vs. real data. We also demonstrate that for appropriately tuned explicit regularization (e.g., dropout) we can degrade DNN training performance on noise datasets without compromising generalization on real data. Our analysis suggests that the notions of effective capacity which are dataset independent are unlikely to explain the generalization performance of deep networks when trained with gradient based methods because training data itself plays an important role in determining the degree of memorization.
原文 arXiv:1706.05394;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1706.05394v2