SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets
Ann Yuan Google Research \AndDaphne Ippolito Google Research \AndVitaly Nikolaev Google Research \ANDChris Callison-Burch University of Pennsylvania \AndAndy Coenen Google Research \AndSebastian Gehrmann Google Research
Abstract
NLP researchers need more, higher-quality text datasets. Human-labeled datasets are expensive to collect, while datasets collected via automatic retrieval from the web such as WikiBio [32] are noisy and can include undesired biases. Moreover, data sourced from the web is often included in datasets used to pretrain models, leading to inadvertent cross-contamination of training and test sets. In this work we introduce a novel method for efficient dataset curation: we use a large language model to provide seed generations to human raters, thereby changing dataset authoring from a writing task to an editing task. We use our method to curate SynthBio–a new evaluation set for WikiBio– composed of structured attribute lists describing fictional individuals, mapped to natural language biographies. We show that our dataset of fictional biographies is less noisy than WikiBio, and also more balanced with respect to gender and nationality.111The dataset can be found at https://storage.googleapis.com/gem-benchmark/SynthBio.json.
中文速览
自然语言生成领域长期缺乏高质量标注数据集，人工标注成本高，而从网络自动抓取的数据（如WikiBio）则噪声大、含偏见，还容易与预训练模型的训练数据交叉污染。为此，研究者提出了一种"大语言模型生成初稿、人工标注者编辑修改"的混合数据集构建流程，将费力的从头写作任务转变为效率更高的编辑任务。基于这一方法，他们构建了SynthBio——一个由2249个虚构人物信息框及4692篇对应传记组成的WikiBio评测集，在性别和国籍分布上比原始WikiBio更加均衡，且传记内容与信息框的事实一致性（faithfulness）也更高。由于数据全部描述虚构人物，模型无法靠背诵真实世界知识取巧，实验也证实在WikiBio上训练的模型迁移到SynthBio后性能明显下降，说明该数据集能更严格地考验模型真正理解结构化数据并生成忠实文本的能力，为评测数据集的高效、可控构建提供了一条可行路径。
原文 arXiv:2111.06467;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2111.06467v2