Visualizing the Loss Landscape of Neural Nets
Hao Li1, Zheng Xu1, Gavin Taylor2, Christoph Studer3, Tom Goldstein1 1University of Maryland, College Park 2United States Naval Academy 3Cornell University
Abstract
Neural network training relies on our ability to find “good” minimizers of highly non-convex loss functions. It is well-known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effects on the underlying loss landscape, are not well understood. In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple “filter normalization” method that helps us visualize loss function curvature and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture affects the loss landscape, and how training parameters affect the shape of minimizers.
中文速览
神经网络的损失函数（loss function）极其复杂，我们至今不清楚为什么某些网络架构（比如残差连接）更容易训练、某些训练方式找到的最优解泛化能力更强。本文提出了一种叫做"滤波器归一化"（filter normalization）的可视化技巧，通过对随机扰动方向按各滤波器的权重尺度进行归一化，消除了网络权重的尺度不变性带来的视觉误导，从而能够公平地并排比较不同网络、不同训练策略所找到的最优解的"尖锐度"或"平坦度"。借助这一方法和一系列高分辨率二维可视化实验，作者发现：网络越深，损失曲面从近似凸形迅速变得极度混乱，泛化误差随之骤降，而残差连接（skip connections）能有效阻止这种混乱化、促使优化落入更平坦的最优解，从而解释了残差连接对超深网络的必要性；此外还确认了小批量SGD找到的平坦极小值确实比大批量方法找到的尖锐极小值泛化更好。这些可视化发现为理解神经网络的可训练性与泛化行为提供了直观的经验性证据，对网络架构设计和训练策略选择均有实际指导意义。
原文 arXiv:1712.09913;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1712.09913v3