Transferring Inductive Biases through Knowledge Distillation
Samira Abnar1, Mostafa Dehghani2, Willem Zuidema1 1University of Amsterdam, 2Google Brain
Abstract
Having the right inductive biases can be crucial in many tasks or scenarios where data or computing resources are a limiting factor, or where training data is not perfectly representative of the conditions at test time. However, defining, designing and efficiently adapting inductive biases is not necessarily straightforward. In this paper, we explore the power of knowledge distillation for transferring the effect of inductive biases from one model to another. We consider families of models with different inductive biases, LSTMs vs. Transformers and CNNs vs. MLPs, in the context of tasks and scenarios where having the right inductive biases is critical. We study the effect of inductive biases on the solutions the models converge to and investigate how and to what extent the effect of inductive biases is transferred through knowledge distillation, in terms of not only performance but also different aspects of converged solutions.
中文速览
当一个模型天生就有"正确的偏好"（归纳偏置，inductive bias）时，它在数据有限或测试场景与训练场景不完全一致的情况下往往表现更好，但把这种偏好"移植"给结构不同的模型并不容易。研究者提出用知识蒸馏（knowledge distillation）来完成这种转移：让具备合适归纳偏置的教师模型（如LSTM或CNN）通过输出的概率分布来"指导"归纳偏置较弱的学生模型（如Transformer或MLP）训练。实验在两个典型场景中验证了这一想法——英语主谓一致预测任务（LSTM教Transformer）和带扰动的MNIST鲁棒性测试（CNN教MLP）——结果表明，蒸馏后的学生模型不仅在分布内和分布外的测试集上性能显著提升，其内部表示结构、置信度校准乃至训练过程中的损失面探索轨迹都与教师模型趋于相似。这项工作揭示了知识蒸馏的作用远不止压缩模型，它能将一种架构独有的"学习偏好"真正传递给另一种架构，为在推理效率与泛化能力之间寻求更好权衡提供了切实可行的方法。
原文 arXiv:2006.00555;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2006.00555v3