Observational Overfitting in Reinforcement Learning
Xingyou Song∗1 Yiding Jiang†1 Stephen Tu1 Yilun Du∗2 Behnam Neyshabur1
Abstract
A major component of overfitting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP). We provide a general framework for analyzing this scenario, which we use to design multiple synthetic benchmarks from only modifying the observation space of an MDP. When an agent overfits to different observation spaces even if the underlying MDP dynamics is fixed, we term this observational overfitting. Our experiments expose intriguing properties especially with regards to implicit regularization, and also corroborate results from previous works in RL generalization and supervised learning (SL).
中文速览
无模型强化学习（model-free RL）中的智能体常常会把奖励信号误归因于观测画面里与任务无关的"杂牌特征"，比如《索尼克》游戏里的计分板或背景纹理，从而导致过拟合、泛化失败。作者提出了一个统一理论框架，把观测函数拆分为"不变的有效特征"和"随环境参数变化的无关特征"两部分，在固定底层MDP动力学的前提下仅改变观测空间，由此构建出一系列可控的合成基准，将这一现象定义为"观测过拟合"（observational overfitting）。在线性二次调节器（LQR）和经典Gym环境的神经网络实验中，他们发现梯度下降会产生隐式正则化（implicit regularization）效应，倾向于找到范数更小、泛化更好的解，并在CoinRun等真实基准上进一步验证了这一假说。这项工作为厘清RL泛化研究中的各种混淆因素提供了一个干净的分析切入点，也为设计更鲁棒的视觉策略提供了理论依据。
原文 arXiv:1912.02975;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1912.02975v2