Language is Power: Representing States Using Natural Language in Reinforcement Learning
Erez Schwartz, Guy Tennenholtz, Chen Tessler and Shie Mannor
Abstract
Recent advances in reinforcement learning have shown its potential to tackle complex real-life tasks. However, as the dimensionality of the task increases, reinforcement learning methods tend to struggle. To overcome this, we explore methods for representing the semantic information embedded in the state. While previous methods focused on information in its raw form (e.g., raw visual input), we propose to represent the state using natural language. Language can represent complex scenarios and concepts, making it a favorable candidate for representation. Empirical evidence, within the domain of ViZDoom, suggests that natural language based agents are more robust, converge faster and perform better than vision based agents, showing the benefit of using natural language representations for reinforcement learning.
中文速览
强化学习（RL）在面对高维复杂任务时往往力不从心，如何为智能体提供一种更有效的状态表示是关键瓶颈。研究者提出用自然语言描述游戏状态，替代以往直接输入原始像素图像或语义分割图的做法：先用一个自定义的自然语言生成器将每一帧画面转换成描述场景的文字句子，再借助词向量（Word2Vec/GloVe）和文本卷积网络（Text-CNN）提取特征，最后送入DQN或PPO等经典强化学习算法训练。在ViZDoom三维射击游戏环境的多个难度场景上，基于自然语言表示的智能体不仅收敛更快、最终得分更高，还对环境变化表现出更强的迁移鲁棒性，优于原始视觉输入和语义分割两类基线方案。这项工作揭示了自然语言不仅是人类之间高效传递信息的媒介，同样可以作为强化学习中语义状态表示的强力选项，为构建可解释、易迁移的智能体开辟了新思路。
原文 arXiv:1910.02789;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1910.02789v2