Differentially Private Fine-tuning of Language Models
Da Yu Sun Yat-sen University. Work was done while an intern at Microsoft Research Asia. Saurabh Naik Microsoft. {snaik, Arturs Backurs Microsoft Research. {arturs.backurs, sigopi, huseyin.inan, jakul, amonteiroman, Sivakanth Gopi Huseyin A. Inan Gautam Kamath Cheriton School of Computer Science, University of Waterloo. Supported by an NSERC Discovery Grant. Janardhan Kulkarni Yin Tat Lee University of Washington and Microsoft Research. Andre Manoel Lukas Wutschitz Sergey Yekhanin Huishuai Zhang Microsoft Research Asia.
Abstract
We give simpler, sparser, and faster algorithms for differentially private fine-tuning of large-scale pre-trained language models, which achieve the state-of-the-art privacy versus utility tradeoffs on many standard NLP tasks. We propose a meta-framework for this problem, inspired by the recent success of highly parameter-efficient methods for fine-tuning. Our experiments show that differentially private adaptations of these approaches outperform previous private algorithms in three important dimensions: utility, privacy, and the computational and memory cost of private training. On many commonly studied datasets, the utility of private models approaches that of non-private models. For example, on the MNLI dataset we achieve an accuracy of $87.8\%$ using RoBERTa-Large and $83.5\%$ using RoBERTa-Base with a privacy budget of $\varepsilon=6.7$ . In comparison, absent privacy constraints, RoBERTa-Large achieves an accuracy of $90.2\%$ . Our findings are similar for natural language generation tasks. Privately fine-tuning with DART, GPT-2-Small, GPT-2-Medium, GPT-2-Large, and GPT-2-XL achieve BLEU scores of 38.5, 42.0, 43.1, and 43.8 respectively (privacy budget of $\varepsilon=6.8,\de
中文速览
大规模预训练语言模型在实际部署时往往需要在敏感数据上做微调，如何在保护用户隐私的同时维持模型性能，是个棘手难题。本文提出了一套基于参数高效微调（parameter-efficient fine-tuning）的差分隐私（differential privacy, DP）训练框架：只对少量新增参数（如 LoRA 低秩矩阵或 Adapter 模块）施加 DP 噪声，而冻结绝大多数预训练权重，从而大幅减少噪声注入量、降低显存开销，并防止隐私训练"遗忘"预训练知识。实验表明，该方法在多项标准 NLP 任务上显著超越此前最佳私有训练方案，例如以隐私预算 ε=6.7 私有微调 RoBERTa-Large 后，在 MNLI 数据集上达到 87.8% 的准确率，与无隐私约束的 90.2% 仅差 2.4 个百分点；此外还发现一个重要规律：模型越大，隐私带来的精度损失越小。这项工作为在严格隐私保证下高效利用大型预训练模型提供了切实可行的路径，对医疗、金融等隐私敏感场景具有重要实用价值。
原文 arXiv:2110.06500;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.06500v2