Taming the Noise in Reinforcement Learning via Soft Updates
Roy Fox∗ Hebrew University、Ari Pakman∗ Columbia University、Naftali Tishby Hebrew University ∗These authors contributed equally to this work.
Abstract
Model-free reinforcement learning algorithms, such as Q-learning, perform poorly in the early stages of learning in noisy environments, because much effort is spent unlearning biased estimates of the state-action value function. The bias results from selecting, among several noisy estimates, the apparent optimum, which may actually be suboptimal. We propose G-learning, a new off-policy learning algorithm that regularizes the value estimates by penalizing deterministic policies in the beginning of the learning process. We show that this method reduces the bias of the value-function estimation, leading to faster convergence to the optimal value and the optimal policy. Moreover, G-learning enables the natural incorporation of prior domain knowledge, when available. The stochastic nature of G-learning also makes it avoid some exploration costs, a property usually attributed only to on-policy algorithms. We illustrate these ideas in several examples, where G-learning results in significant improvements of the convergence rate and the cost of the learning process.
中文速览
Q-learning在嘈杂环境中学习缓慢，根本原因在于它用"取最小值"这个操作来更新价值估计，噪声会让这个最小值产生系统性的乐观偏差，导致算法在早期就错误地"押注"某些次优动作，之后还要花大量时间纠正。论文提出的G-learning通过引入一个信息论惩罚项，在学习初期让策略保持随机性（贴近先验策略），随着样本积累再逐渐向确定性贪心策略过渡，本质上用"软最小值"替代了原来的"硬最小值"，从而大幅压低了价值函数估计的偏差。理论上证明了该算法收敛，实验中也展示出更快的收敛速度和更低的学习过程代价，而且G-learning还天然能够感知探索成本——这种特性此前只在SARSA等在策略算法中才能看到。这项工作既为强化学习的噪声问题提供了有理论保障的解法，又提供了一个将先验领域知识自然融入无模型学习的统一框架，实用价值显著。
原文 arXiv:1512.08562;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1512.08562v4