Towards Learning Convolutions from Scratch
Behnam Neyshabur Google
Abstract
Convolution is one of the most essential components of architectures used in computer vision. As machine learning moves towards reducing the expert bias and learning it from data, a natural next step seems to be learning convolution-like structures from scratch. This, however, has proven elusive. For example, current state-of-the-art architecture search algorithms use convolution as one of the existing modules rather than learning it from data. In an attempt to understand the inductive bias that gives rise to convolutions, we investigate minimum description length as a guiding principle and show that in some settings, it can indeed be indicative of the performance of architectures. To find architectures with small description length, we propose $\beta$ -lasso, a simple variant of lasso algorithm that, when applied on fully-connected networks for image classification tasks, learns architectures with local connections and achieves state-of-the-art accuracies for training fully-connected nets on CIFAR-10 (85.19%), CIFAR-100 (59.56%) and SVHN (94.07%) bridging the gap between fully-connected and convolutional nets.
中文速览
卷积（convolution）是计算机视觉深度学习的核心组件，但它本质上是人工设计的先验知识——能不能让网络自己从数据中"发现"卷积结构，一直是个难题。为了弄清楚卷积为何有效，研究者引入最小描述长度（Minimum Description Length，MDL）作为理论框架，发现参数越少、结构越稀疏的网络往往泛化越好。基于此，他们提出了一种叫做 β-lasso 的简单训练算法——它是经典 lasso 正则化的变体，通过更激进的软阈值处理迫使全连接网络自动学出局部连接的稀疏结构。实验结果显示，β-lasso 在全连接网络上分别达到了 CIFAR-10 85.19%、CIFAR-100 59.56%、SVHN 94.07% 的准确率，大幅刷新了全连接网络的记录，并将其与卷积网络之间的性能差距缩小到前所未有的程度。这项工作的意义在于：它不仅从理论上解释了局部连接才是卷积最关键的归纳偏置，还提供了一条无需人工指定卷积结构、让网络自己学出类卷积特性的可行路径。
原文 arXiv:2007.13657;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2007.13657v1