FLAME: Free-form Language-based Motion Synthesis & Editing
Jihoon Kim,1, 2 Jiseob Kim, 2 Sungjoon Choi 1
Abstract
Text-based motion generation models are drawing a surge of interest for their potential for automating the motion-making process in the game, animation, or robot industries. In this paper, we propose a diffusion-based motion synthesis and editing model named FLAME. Inspired by the recent successes in diffusion models, we integrate diffusion-based generative models into the motion domain. FLAME can generate high-fidelity motions well aligned with the given text. Also, it can edit the parts of the motion, both frame-wise and joint-wise, without any fine-tuning. FLAME involves a new transformer-based architecture we devise to better handle motion data, which is found to be crucial to manage variable-length motions and well attend to free-form text. In experiments, we show that FLAME achieves state-of-the-art generation performances on three text-motion datasets: HumanML3D, BABEL, and KIT. We also demonstrate that FLAME’s editing capability can be extended to other tasks such as motion prediction or motion in-betweening, which have been previously covered by dedicated models.
中文速览
用扩散模型（diffusion model）来自动生成和编辑3D人体动作一直很有挑战性，因为动作数据既有时序性又有长度可变的特点，主流方法难以灵活地结合自然语言描述进行控制。FLAME提出了一个基于扩散模型的统一框架，专门为动作数据设计了新的Transformer解码器架构，借助预训练语言模型理解自由形式的文本输入，并通过无分类器引导（classifier-free guidance）实现文本与动作的精准对齐。在不需要任何微调的情况下，同一个训练好的模型既能从文本描述直接合成动作，也能对已有动作按帧或按关节进行局部编辑，还能自然扩展到动作预测和动作插值等传统任务。实验表明，FLAME在HumanML3D、BABEL、KIT三个文本-动作数据集上均达到当时最优性能，是首个将扩散模型引入动作合成与编辑领域的工作，为游戏、动画和机器人行业的动作自动化制作提供了一个灵活且强大的统一方案。
原文 arXiv:2209.00349;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2209.00349v2