Director: Generator-Classifiers For Supervised Language Modeling
Kushal Arora McGill University Mila、Kurt Shuster Meta AI、Sainbayar Sukhbaatar Meta AI、Jason Weston Meta AI Work done during an internship at Meta AI.
Abstract
Current language models achieve low perplexity but their resulting generations still suffer from toxic responses, repetitiveness, and contradictions. The standard language modeling setup fails to address these issues. In this paper, we introduce a new architecture, Director, that consists of a unified generator-classifier with both a language modeling and a classification head for each output token. Training is conducted jointly using both standard language modeling data, and data labeled with desirable and undesirable sequences. Experiments in several settings show that the model has competitive training and decoding speed compared to standard language models while yielding superior results, avoiding undesirable behaviors while maintaining generation quality. It also outperforms existing model-guiding approaches in terms of both accuracy and efficiency. Our code is made publicly available111https://parl.ai/projects/director.
中文速览
大语言模型虽然困惑度很低，但生成内容仍频繁出现有毒言论、自相矛盾和无意义重复等问题，根本原因在于标准语言模型训练只能利用"正样本"、无法从带有负标签的监督数据中学习。为此，研究者提出了 Director 这一统一的生成器-分类器架构：在标准 Transformer 解码器的基础上，为每个输出位置同时挂载一个语言模型头和一个分类头，前者用无标签数据训练，后者用标注了好/坏序列的有监督数据训练，解码时将两个头的输出相乘并归一化，就能在逐词生成的同时自动规避不良属性。实验覆盖毒性、矛盾和重复三类任务，结果显示 Director 在生成质量和安全性上均优于重排序（reranking）、FUDGE 和 PACER 等现有方法，且由于分类头与语言模型头共享同一解码器核心、对词表中所有候选词并行打分，训练和推理速度几乎与普通语言模型持平。这项工作的意义在于提供了一个简单、高效、通用的框架，让语言模型能够直接从负样本中学习，从而在不牺牲速度的前提下显著提升生成内容的安全性与可控性。
原文 arXiv:2206.07694;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2206.07694v2