Imitation Learning as f-Divergence Minimization
Liyiming Ke 11 Sanjiban Choudhury 11 Matt Barnes 11 Wen Sun 22 Gilwoo Lee 11 Siddhartha Srinivasa 11
Abstract
We address the problem of imitation learning with multi-modal demonstrations. Instead of attempting to learn all modes, we argue that in many tasks it is sufficient to imitate any one of them. We show that the state-of-the-art methods such as GAIL and behavior cloning, due to their choice of loss function, often incorrectly interpolate between such modes. Our key insight is to minimize the right divergence between the learner and the expert state-action distributions, namely the reverse KL divergence or I-projection. We propose a general imitation learning framework for estimating and minimizing any $f$ -Divergence. By plugging in different divergences, we are able to recover existing algorithms such as Behavior Cloning (Kullback-Leibler), GAIL (Jensen Shannon) and DAgger (Total Variation). Empirical results show that our approximate I-projection technique is able to imitate multi-modal behaviors more reliably than GAIL and behavior cloning.
中文速览
模仿学习中有个长期被忽视的麻烦：当专家示范存在多种截然不同的行为模式时（比如绕障碍物既可以左转也可以右转），现有主流方法如行为克隆和GAIL会因为使用了KL散度或JS散度这类"模式覆盖"型损失函数，错误地在多个模式之间取平均，导致学习者做出两边都不靠的危险动作。研究者将模仿学习统一建模为学习者与专家状态-动作分布之间的f-散度最小化问题，并指出只需将散度换成反向KL散度（即I-projection），就能让策略自然地"吸附"到某一个模式上而非在模式间插值。他们据此提出了通用框架f-VIM，既可以用不同散度复现行为克隆、GAIL、DAgger等已有算法，也可以推导出针对反向KL的高效近似算法，并在多种模拟环境中验证了其比GAIL和行为克隆更可靠地学到单一一致模式。这项工作的意义在于，它不仅提供了理解现有算法失败原因的统一理论视角，还给出了一个简单而实用的修复方案——换对散度，就能让模仿学习在多模态示范下更安全、更稳定。
原文 arXiv:1905.12888;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1905.12888v2