Cross Entropy in Deep Learning of Classifiers Is Unnecessary - ISBE Error is All You Need
Władysław Skarbek111 Author is with the Faculty of Electronics and Information Technology, Warsaw University of Technology, email:
Abstract
In deep learning classifiers, the cost function usually takes the form of a combination of SoftMax and CrossEntropy functions. The SoftMax unit transforms the scores predicted by the model network into assessments of the degree (probabilities) of an object’s membership to a given class. On the other hand, CrossEntropy measures the divergence of this prediction from the distribution of target scores. This work introduces the ISBE functionality, justifying the thesis about the redundancy of cross entropy computation in deep learning of classifiers. Not only can we omit the calculation of entropy, but also, during back-propagation, there is no need to direct the error to the normalization unit for its backward transformation. Instead, the error is sent directly to the model’s network. Using examples of perceptron and convolutional networks as classifiers of images from the MNIST collection, it is observed for ISBE that results are not degraded with SoftMax only, but also with other activation functions such as Sigmoid, Tanh, or their hard variants HardSigmoid and HardTanh. Moreover, up to three percent of time is saved within the total time of forward and backward stages. The article
中文速览
深度学习分类器通常把SoftMax和交叉熵（CrossEntropy）捆绑在一起作为损失函数，但这套组合在计算和反向传播时存在一定冗余。本文提出一种名为ISBE的简化方案：在前向推理阶段只做归一化（把原始分数转成软分数），在反向传播阶段直接把"预测软分数与目标分数之差"作为误差送回网络，完全跳过交叉熵的计算以及SoftMax的反向变换。理论上，当归一化函数为SoftMax时，ISBE传回的梯度与完整的CrossEntropy∘SoftMax完全一致；实验在MNIST手写数字数据集上进一步验证，SoftMax、Sigmoid、Tanh及其硬版本HardSigmoid、HardTanh均能保持99.5%以上的分类准确率，同时前向+反向总耗时节省约3%。这一结论说明交叉熵的显式计算在分类训练中其实是多余的，工程上可以放心省掉，既简化代码又略微加速训练。
原文 arXiv:2311.16357;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2311.16357v1