Muesli: Combining Improvements in Policy Optimization
Matteo Hessel Ivo Danihelka Fabio Viola Arthur Guez Simon Schmitt Laurent Sifre Theophane Weber David Silver Hado van Hasselt
Abstract
We propose a novel policy update that combines regularized policy optimization with model learning as an auxiliary loss. The update (henceforth Muesli) matches MuZero’s state-of-the-art performance on Atari. Notably, Muesli does so without using deep search: it acts directly with a policy network and has computation speed comparable to model-free baselines. The Atari results are complemented by extensive ablations, and by additional results on continuous control and 9x9 Go.
中文速览
强化学习领域一直存在一个两难困境：MuZero靠深度树搜索（MCTS）在Atari游戏上拿到顶尖成绩，但搜索过程计算开销巨大；而不用搜索的无模型方法又很难达到同等水平。Muesli把"正则化策略优化"和"学习环境模型"巧妙地结合起来——用类似MuZero的单步前向模型来估计动作价值，再通过一种改进的裁剪MPO机制把这些价值折算成策略更新目标，整个过程直接用策略网络行动，完全不依赖深度搜索。实验结果显示，Muesli在57款Atari经典游戏上达到了与MuZero相当的最优水平，计算速度却与普通无模型算法相近，同时在连续控制任务和9×9围棋自弈中也表现出良好的泛化能力。这项工作证明，把模型学习作为辅助损失融入策略优化，就能在不牺牲计算效率的前提下获得媲美规划方法的性能，为设计兼顾效果与效率的强化学习算法提供了新思路。
原文 arXiv:2104.06159;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2104.06159v2