Convolutional Neural Networks combined with Runge–Kutta Methods
\fnmMai \surZhu \fnmBo \surChang * [
Abstract
A convolutional neural network can be constructed using numerical methods for solving dynamical systems, since the forward pass of the network can be regarded as a trajectory of a dynamical system. However, existing models based on numerical solvers cannot avoid the iterations of implicit methods, which makes the models inefficient at inference time. In this paper, we reinterpret the pre-activation Residual Networks (ResNets) and their variants from the dynamical systems view. We consider that the iterations of implicit Runge–Kutta methods are fused into the training of these models. Moreover, we propose a novel approach to constructing network models based on high-order Runge–Kutta methods in order to achieve higher efficiency. Our proposed models are referred to as the Runge–Kutta Convolutional Neural Networks (RKCNNs). The RKCNNs are evaluated on multiple benchmark datasets. The experimental results show that RKCNNs are vastly superior to other dynamical system network models: they achieve higher accuracy with much fewer resources. They also expand the family of network models based on numerical methods for dynamical systems.
中文速览
用数值方法中的龙格–库塔（Runge-Kutta，RK）方法来构建卷积神经网络时，现有模型（如ODE-Net）在推理阶段仍需反复迭代求解隐式方程，导致计算效率低下。作者重新解释了预激活残差网络（pre-activation ResNet）的残差映射：它并非只对应最简单的前向欧拉方法，而是在整体上近似了某种RK方法（包括隐式RK方法），迭代过程已被"融合"进网络的训练之中，从而彻底消除了推理时的独立迭代开销。在此基础上，作者提出了龙格–库塔卷积神经网络（RKCNN），将RK方法的每个计算阶段分别用独立子网络来近似，使网络结构更精确地体现高阶RK方法的细节。在MNIST、SVHN和CIFAR等多个图像分类基准上，RKCNN以更少的参数和计算资源取得了比同类数值方法网络更高的精度，为基于动力系统数值方法的神经网络设计提供了一条兼顾精度与效率的新路径。
原文 arXiv:1802.08831;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1802.08831v7