arXiv:2004.04136 · 中英对照阅读
CURL: Contrastive Unsupervised Representations for Reinforcement Learning
中文速览
像素画面信息量大,导致强化学习要反复试很多次才能学会控制,效率远不如直接读取状态数据。CURL把对比学习(contrastive learning)和强化学习结合起来,让智能体在练习控制的同时,学会辨认画面中对任务有用的信息,再用这些特征做决策。实验显示,它在控制任务和电子游戏上都比此前的像素方法更省交互数据;在一组连续控制任务中,表现已接近使用状态数据的方法。这个结果说明,经过合适的表示学习,智能体有望只看图像也能高效学习,从而减少模拟实验和真实机器人的训练成本。
摘要
We present CURL: Contrastive Unsupervised Representations for Reinforcement Learning. CURL extracts high-level features from raw pixels using contrastive learning and performs off-policy control on top of the extracted features. CURL outperforms prior pixel-based methods, both model-based and model-free, on complex tasks in the DeepMind Control Suite and Atari Games showing 1.9x and 1.2x performance gains at the 100K environment and interaction steps benchmarks respectively. On the DeepMind Control Suite, CURL is the first image-based algorithm to nearly match the sample-efficiency of methods that use state-based features. Our code is open-sourced and available at https://www.github.com/MishaLaskin/curl.
术语表
- CURL
- 对比式无监督强化学习表征
- Contrastive Learning
- 对比学习
- Unsupervised Representation Learning
- 无监督表征学习
- Reinforcement Learning (RL)
- 强化学习
- Sample Efficiency
- 样本效率
- Pixel-Based Methods
- 基于像素的方法
- High-Dimensional Observations
- 高维观测
- Raw Pixels
- 原始像素
- Off-Policy Control
- 离策略控制
- Model-Based Reinforcement Learning
- 基于模型的强化学习
- Model-Free Reinforcement Learning
- 无模型强化学习
- Auxiliary Tasks
- 辅助任务
- Self-Supervised Learning
- 自监督学习
- Semantic Representation
- 语义表征
- World Model
- 世界模型
- Predictive Model
- 预测模型
- Contrastive Objective
- 对比目标
- Data Augmentation
- 数据增强
- Temporal Frame Stack
- 时序帧堆叠
- Soft Actor-Critic (SAC)
- 软演员-评论家算法
- Dreamer
- Dreamer
- DeepMind Control Suite (DMControl)
- DeepMind 控制套件
- Atari Games
- Atari 游戏
- Rainbow DQN
- Rainbow DQN
- Efficient Rainbow
- Efficient Rainbow
- SimPLe
- SimPLe
- Contrastive Predictive Coding (CPC)
- 对比预测编码
- State-Based Features
- 基于状态的特征