TorchBeast: A PyTorch Platform for Distributed RL
Heinrich Küttler Correspondence to Facebook AI Research Nantas Nardelli Facebook AI Research University of Oxford Thibaut Lavril Facebook AI Research Marco Selvatici Facebook AI Research Imperial College London Viswanath Sivakumar Facebook AI Research Tim Rocktäschel Facebook AI Research University College London Edward Grefenstette Facebook AI Research University College London
Abstract
TorchBeast is a platform for reinforcement learning (RL) research in PyTorch. It implements a version of the popular IMPALA algorithm [1] for fast, asynchronous, parallel training of RL agents. Additionally, TorchBeast has simplicity as an explicit design goal: We provide both a pure-Python implementation (“MonoBeast”) as well as a multi-machine high-performance version (“PolyBeast”). In the latter, parts of the implementation are written in C++, but all parts pertaining to machine learning are kept in simple Python using PyTorch [2], with the environments provided using the OpenAI Gym interface [3]. This enables researchers to conduct scalable RL research using TorchBeast without any programming knowledge beyond Python and PyTorch. In this paper, we describe the TorchBeast design principles and implementation and demonstrate that it performs on-par with IMPALA on Atari. TorchBeast is released as an open-source package under the Apache 2.0 license and is available at https://github.com/facebookresearch/torchbeast.
中文速览
强化学习（RL）研究长期缺乏易用、高性能且可扩展的开源实现，导致大量工作只能在少数拥有专业工程能力的机构中推进。TorchBeast 为此提供了一个基于 PyTorch 的 IMPALA 算法实现平台，包含纯 Python 的轻量版本 MonoBeast 和借助 C++ 与 gRPC 支持多机分布式训练的高性能版本 PolyBeast，但所有机器学习相关代码均保留在 Python 层，研究者无需掌握 C++ 即可灵活修改模型和环境。实验表明，TorchBeast 在 Atari 游戏基准上的训练性能和吞吐量与 DeepMind 官方 TensorFlow IMPALA 实现不相上下。这一工作的价值在于大幅降低了分布式强化学习研究的门槛，让更多研究者能够在 PyTorch 生态中快速复现、改进并拓展规模化 RL 方法。
原文 arXiv:1910.03552;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1910.03552v1