A Distributional Approach to Controlled Text Generation
Muhammad Khalifa Cairo University、Hady Elsahar Naver Labs Europe、Marc Dymetman Naver Labs Europe \AND Equal Contributions.Work done during an internship at NAVER Labs Europe.
Abstract
We propose a Distributional Approach for addressing Controlled Text Generation from pre-trained Language Models (LMs). This approach permits to specify, in a single formal framework, both “pointwise” and “distributional” constraints over the target LM — to our knowledge, the first model with such generality — while minimizing KL divergence from the initial LM distribution. The optimal target distribution is then uniquely determined as an explicit EBM (Energy-Based Model) representation. From that optimal representation we then train a target controlled Autoregressive LM through an adaptive distributional variant of Policy Gradient. We conduct a first set of experiments over pointwise constraints showing the advantages of our approach over a set of baselines, in terms of obtaining a controlled LM balancing constraint satisfaction with divergence from the initial LM. We then perform experiments over distributional constraints, a unique feature of our approach, demonstrating its potential as a remedy to the problem of Bias in Language Models. Through an ablation study, we show the effectiveness of our adaptive technique for obtaining faster convergence.111Code available at https://git
中文速览
预训练语言模型（如GPT-2）虽然生成质量极高，却难以按需控制——比如让它避免偏见或聚焦特定话题，尤其是当我们希望对"整体生成分布"施加统计层面的约束时，现有方法几乎无能为力。论文提出了GDC（Generation with Distributional Control）框架，将可控文本生成统一表述为一个约束满足加KL散度最小化的优化问题：既支持"逐样本"的硬性约束（每条生成都必须满足某条件），也支持"分布式"约束（例如要求50%的生成提到女性角色），理论上可以证明最优目标分布唯一存在，且具有能量模型（Energy-Based Model，EBM）的解析形式；在此基础上，论文进一步提出KL自适应分布策略梯度算法（KL-Adaptive DPG），将该EBM蒸馏为可直接采样的自回归语言模型。实验表明，GDC在话题控制、情感控制等常规任务上优于多个基线，同时在缓解语言模型性别偏见等分布层面的问题上展现出独特潜力，为解决预训练模型的社会偏见问题提供了一条有理论保障的新路径。
原文 arXiv:2012.11635;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2012.11635v2