Convolutional Neural Networks Analyzed via Convolutional Sparse Coding
\nameVardan Papyan* \addrDepartment of Computer Science Technion -– Israel Institute of Technology Technion City, Haifa 32000, Israel \AND\nameYaniv Romano* \addrDepartment of Electrical Engineering Technion –- Israel Institute of Technology Technion City, Haifa 32000, Israel \AND\nameMichael Elad \addrDepartment of Computer Science Technion –- Israel Institute of Technology Technion City, Haifa 32000, Israel The authors contributed equally to this work.
Abstract
Convolutional neural networks (CNN) have led to many state-of-the-art results spanning through various fields. However, a clear and profound theoretical understanding of the forward pass, the core algorithm of CNN, is still lacking. In parallel, within the wide field of sparse approximation, Convolutional Sparse Coding (CSC) has gained increasing attention in recent years. A theoretical study of this model was recently conducted, establishing it as a reliable and stable alternative to the commonly practiced patch-based processing. Herein, we propose a novel multi-layer model, ML-CSC, in which signals are assumed to emerge from a cascade of CSC layers. This is shown to be tightly connected to CNN, so much so that the forward pass of the CNN is in fact the thresholding pursuit serving the ML-CSC model. This connection brings a fresh view to CNN, as we are able to attribute to this architecture theoretical claims such as uniqueness of the representations throughout the network, and their stable estimation, all guaranteed under simple local sparsity conditions. Lastly, identifying the weaknesses in the above pursuit scheme, we propose an alternative to the forward pass, which is connec
中文速览
卷积神经网络（CNN）在各领域屡创佳绩，却始终缺乏一套扎实的数学理论来解释它"为什么管用"。作者从稀疏表示领域的卷积稀疏编码（Convolutional Sparse Coding, CSC）出发，提出了一个多层模型 ML-CSC：假设信号由一系列 CSC 层级联生成，每一层的表示向量本身又满足卷积稀疏结构。通过严格的数学推导，作者证明 CNN 的前向传播（forward pass）恰好等价于 ML-CSC 模型的逐层阈值追踪算法，从而首次为 CNN 赋予了清晰的数学含义——只要各层的稀疏表示在局部意义上足够稀疏，前向传播就能唯一且稳定地恢复出各层的隐含表示，输入受有界噪声干扰时输出扰动同样有界。在此基础上，作者还提出了一种更优的替代追踪方案（逐层基追踪），并将其与反卷积网络、循环网络和残差网络建立了联系，附有更强的理论保证。这项工作首次将深度学习与稀疏表示理论紧密统一，为理解 CNN 的工作机制提供了坚实的理论基础。
原文 arXiv:1607.08194;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1607.08194v4