Deduplicating Training Data Mitigates Privacy Risks in Language Models
Nikhil Kandpal Eric Wallace Colin Raffel
Abstract
Past work has shown that large language models are susceptible to privacy attacks, where adversaries generate sequences from a trained model and detect which sequences are memorized from the training set. In this work, we show that the success of these attacks is largely due to duplication in commonly used web-scraped training sets. We first show that the rate at which language models regenerate training sequences is superlinearly related to a sequence’s count in the training set. For instance, a sequence that is present 10 times in the training data is on average generated ${\sim}1000\times$ more often than a sequence that is present only once. We next show that existing methods for detecting memorized sequences have near-chance accuracy on non-duplicated training sequences. Finally, we find that after applying methods to deduplicate training data, language models are considerably more secure against these types of privacy attacks. Taken together, our results motivate an increased focus on deduplication in privacy-sensitive applications and a reevaluation of the practicality of existing privacy attacks.
中文速览
大型语言模型（LLM）存在隐私泄露风险——攻击者可以让模型"吐出"训练数据，但究竟是什么让这类攻击得以奏效，过去并不清楚。研究者发现，根本原因在于训练数据集中大量存在的重复序列：一段文本在训练集里出现10次，模型重新生成它的频率平均比只出现1次的文本高出约1000倍，两者呈超线性关系。进一步实验表明，现有的成员推断（membership inference）检测方法对那些在训练集中只出现一次的序列几乎毫无识别能力，准确率接近随机猜测；而对训练数据进行序列级去重（deduplication）后，模型泄露的训练数据量减少约20倍，攻击有效性大幅下降。这一发现说明，过去对语言模型隐私攻击威胁的评估很可能因训练数据的高重复率而被严重高估，在隐私敏感场景下对训练数据做彻底去重才是抵御此类攻击最直接有效的手段。
原文 arXiv:2202.06539;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2202.06539v3