Abstract

Offline reinforcement learning (RL) has shown promise in learning policies from a prerecorded dataset without any interaction with the environment, and has the potential to significantly improve the scalability and safety of policy learning. Recent advances in offline model-based RL have further improved previous offline model-free approaches by enabling greater generalization to new states. These advantages make offline model-based RL particularly appealing for learning complex skills from raw sensor observations, such as images, since learning visuomotor policies using deep RL methods is much more sample-inefficient and unsafe (e.g. failure to generalize to lighting changes) than learning from low-dimensional state inputs. Hence, learning vision-based tasks end-to-end from offline data becomes crucial for robotics and control. However, offline model-based RL from visual inputs is challenging because the offline model-based RL problem critically relies on accurate uncertainty quantification of the model’s predictions to avoid falling off the data distribution and estimating uncertainty of high-dimensional visual dynamics models such as learning an ensemble of video prediction mode