FractalNet: Ultra-Deep Neural Networks without Residuals
Gustav Larsson University of Chicago \AndMichael Maire TTI Chicago \AndGregory Shakhnarovich TTI Chicago
Abstract
We introduce a design strategy for neural network macro-architecture based on self-similarity. Repeated application of a simple expansion rule generates deep networks whose structural layouts are precisely truncated fractals. These networks contain interacting subpaths of different lengths, but do not include any pass-through or residual connections; every internal signal is transformed by a filter and nonlinearity before being seen by subsequent layers. In experiments, fractal networks match the excellent performance of standard residual networks on both CIFAR and ImageNet classification tasks, thereby demonstrating that residual representations may not be fundamental to the success of extremely deep convolutional neural networks. Rather, the key may be the ability to transition, during training, from effectively shallow to deep. We note similarities with student-teacher behavior and develop drop-path, a natural extension of dropout, to regularize co-adaptation of subpaths in fractal architectures. Such regularization allows extraction of high-performance fixed-depth subnetworks. Additionally, fractal networks exhibit an anytime property: shallow subnetworks provide a quick answer
中文速览
残差网络（ResNet）大获成功后，学界普遍认为"让网络学残差"是训练超深网络的关键，但这篇论文用一种基于分形（fractal）的网络结构证明了这一假设并不成立。研究者提出了FractalNet：只需反复套用一条简单的展开规则，就能生成结构上精确对应截断分形的深层网络，其中天然包含长短不一的多条并行子路径，却完全没有任何直通（pass-through）或残差连接。在CIFAR和ImageNet图像分类任务上，FractalNet的表现与ResNet不相上下，说明残差表示并非超深网络成功的根本原因，真正关键的是训练过程中网络能够从"实际较浅"逐渐过渡到"实际较深"。此外，论文还提出了drop-path这一正则化方法——类比dropout对激活值的处理，drop-path随机丢弃并行路径，防止子路径之间的共适应，从而让每条子网络都能独立给出可靠预测，并赋予整个网络"随时可查询"（anytime）的特性：浅层子网络能快速给出粗略答案，深层子网络则以更长延迟换取更高精度。这项工作不仅为超深网络设计提供了一条简洁的新路径，也从机理上重新审视了残差学习、深度监督、师生网络等一系列被视为各自独立的设计技巧之间的内在联系。
原文 arXiv:1605.07648;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1605.07648v4