Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
Yiping Lu School of Mathematical Sciences Peking University, Beijing, China、Aoxiao Zhong, Quanzheng Li MGH/BWH Center for Clinical Data Science Department of Radiology, Massachusetts General Hospital Harvard Medical School、Bin Dong Beijing International Center for Mathematical Research, Peking University Center for Data Science, Peking University Beijing Institute of Big Data Research Beijing, China
Abstract
Deep neural networks have become the state-of-the-art models in numerous machine learning tasks. However, general guidance to network architecture design is still missing. In our work, we bridge deep neural network design with numerical differential equations. We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations. This finding brings us a brand new perspective on the design of effective deep architectures. We can take advantage of the rich knowledge in numerical analysis to guide us in designing new and potentially more effective deep networks. As an example, we propose a linear multi-step architecture (LM-architecture) which is inspired by the linear multi-step method solving ordinary differential equations. The LM-architecture is an effective structure that can be used on any ResNet-like networks. In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. the networks obtained by applying the LM-architecture on ResNet and ResNeXt respectively) can achieve noticeably higher accuracy than ResNet and ResNeXt on both CIFAR and ImageNet with comparable numbers of
中文速览
深度神经网络的架构设计长期依赖经验摸索，缺乏系统性理论指导，而这项研究发现了一条新路：把网络架构和数值微分方程（numerical differential equations）对应起来。作者证明了ResNet、PolyNet、FractalNet、RevNet等主流网络本质上是求解常微分方程（ODE）的不同数值离散格式——例如ResNet对应前向欧拉法、PolyNet对应后向欧拉法、FractalNet对应龙格-库塔法。受ODE数值解中"线性多步法（linear multi-step method）"的启发，作者提出了一种新的网络结构LM-architecture，将其应用于ResNet和ResNeXt后，在CIFAR和ImageNet上以相近参数量取得了明显更高的精度，并且能在压缩超过50%网络规模的同时保持相当的性能。这一工作的意义在于：它把数值分析中数百年积累的理论工具直接引入深度网络设计，让网络架构的选择从"靠直觉"变成"有依据"，为未来设计更高效的深度网络提供了一套可操作的理论框架。
原文 arXiv:1710.10121;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1710.10121v3