BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems
Zachary Lipton⋆ Xiujun Li† Jianfeng Gao† Lihong Li‡ Faisal Ahmed† Li Deng§∗ ⋆Carnegie Mellon University, Pittsburgh, PA, USA ⋆Amazon AI, Palo Alto, CA, USA †Microsoft Research, Redmond, WA, USA ‡Google Inc., Kirkland, WA, USA §Citadel, Seattle, WA, USA This work was done while ZL, LL、LD were with Microsoft.
Abstract
We present a new algorithm that significantly improves the efficiency of exploration for deep Q-learning agents in dialogue systems. Our agents explore via Thompson sampling, drawing Monte Carlo samples from a Bayes-by-Backprop neural network. Our algorithm learns much faster than common exploration strategies such as $\epsilon$ -greedy, Boltzmann, bootstrapping, and intrinsic-reward-based ones. Additionally, we show that spiking the replay buffer with experiences from just a few successful episodes can make Q-learning feasible when it might otherwise fail.
中文速览
对话系统中智能体要靠反复试错来学会订票等任务，但奖励稀疏、动作空间庞大，导致常见的随机探索策略（如ε-greedy）往往要跑几千轮才能见到一次成功。本文提出了一种基于贝叶斯神经网络的深度Q网络算法BBQN（Bayes-by-Backprop Q-Network），通过对网络权重维护概率分布来量化不确定性，再用汤普森采样（Thompson Sampling）在不确定性高的状态-动作区域优先探索，从而大幅提升探索效率。此外，作者还提出"经验回放缓冲区预填充"（Replay Buffer Spiking）技巧，仅用少量来自规则系统的成功对话提前充填经验池，使Q学习在几乎从零开始的情况下也能顺利启动。在静态和动态扩展两种订票对话任务上，BBQN的学习速度和最终成功率均显著优于ε-greedy、Boltzmann探索、Bootstrapping及内在奖励等多种基线方法，真实用户评测也验证了这一优势，为稀疏奖励对话系统的高效探索提供了一套实用方案。
原文 arXiv:1608.05081;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1608.05081v4