Reducing Neural Network Parameter Initialization Into an SMT Problem
Mohamad H. Danesh
Abstract
Training a neural network (NN) depends on multiple factors, including but not limited to the initial weights. In this paper, we focus on initializing deep NN parameters such that it performs better, comparing to random or zero initialization. We do this by reducing the process of initialization into an SMT solver. Previous works consider certain activation functions on small NNs, however the studied NN is a deep network with different activation functions. Our experiments show that the proposed approach for parameter initialization achieves better performance comparing to randomly initialized networks.
中文速览
用可满足性模理论（SMT，Satisfiability Modulo Theories）求解器来为深度神经网络找到一组好的初始参数，是本文要解决的核心问题——传统随机初始化无法保证网络快速收敛到较优解。作者把神经网络的训练过程转化为一个一阶逻辑约束公式，将权重和偏置作为SMT变量，要求网络对训练样本的预测输出满足正确分类的不等式约束，并额外加入非零约束来避免平凡解，从而让SMT求解器直接"算出"一组初始权重。在手写数字识别（MNIST）任务上的实验表明，用SMT求解器得到的初始参数比随机初始化的网络表现更好、收敛更快。这项工作的意义在于，它开创性地将形式化方法引入深度网络的参数初始化，为带有非线性激活函数（如ReLU）的多隐层网络提供了一种有理论保障的初始化思路。
原文 arXiv:2011.01191;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2011.01191v3