Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
Stefan Elfwinga Eiji Uchibea,b Kenji Doyab
Abstract
In recent years, neural networks have enjoyed a renaissance as function approximators in reinforcement learning. Two decades after Tesauro’s TD-Gammon achieved near top-level human performance in backgammon, the deep reinforcement learning algorithm DQN achieved human-level performance in many Atari 2600 games. The purpose of this study is twofold. First, we propose two activation functions for neural network function approximation in reinforcement learning: the sigmoid-weighted linear unit (SiLU) and its derivative function (dSiLU). The activation of the SiLU is computed by the sigmoid function multiplied by its input. Second, we suggest that the more traditional approach of using on-policy learning with eligibility traces, instead of experience replay, and softmax action selection with simple annealing can be competitive with DQN, without the need for a separate target network. We validate our proposed approach by, first, achieving new state-of-the-art results in both stochastic SZ-Tetris and Tetris with a small 10 $\times$ 10 board, using TD( $\lambda$ ) learning and shallow dSiLU network agents, and, then, by outperforming DQN in the Atari 2600 domain by using a deep Sarsa( $\l
中文速览
强化学习（reinforcement learning）长期面临一个难题：如何选择合适的神经网络激活函数，以及如何在不依赖复杂技巧的情况下稳定训练。本文提出了两种新激活函数——S形加权线性单元（SiLU，用sigmoid函数乘以输入值计算激活）及其导数函数（dSiLU），并主张用在线策略学习（on-policy learning）配合资格迹（eligibility traces）和softmax动作选择来替代DQN所依赖的经验回放与独立目标网络。实验结果表明，在随机SZ-俄罗斯方块和10×10标准俄罗斯方块上，dSiLU网络将已有最优分数提升了约20%，创下新纪录；在Atari 2600游戏测试中，使用SiLU和dSiLU的深度Sarsa(λ)智能体在12款游戏上的平均得分分别比DQN和双重DQN高出232%和161%。这项工作说明，经典的在线强化学习思路结合精心设计的激活函数，无需繁复的工程改进就能超越当时的主流深度强化学习方法，为激活函数设计和算法选择提供了新的参考。
原文 arXiv:1702.03118;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1702.03118v3