Compressing GANs using Knowledge Distillation
Angeline Aguinaldo Ping-Yeh Chiang Alex Gain Ameya Patil Kolten Pearson Soheil Feizi
Abstract
Generative Adversarial Networks (GANs) have been used in several machine learning tasks such as domain transfer, super resolution, and synthetic data generation. State-of-the-art GANs often use tens of millions of parameters, making them expensive to deploy for applications in low SWAP (size, weight, and power) hardware, such as mobile devices, and for applications with real time capabilities. There has been no work found to reduce the number of parameters used in GANs. Therefore, we propose a method to compress GANs using knowledge distillation techniques, in which a smaller “student” GAN learns to mimic a larger “teacher” GAN. We show that the distillation methods used on MNIST, CIFAR-10, and Celeb-A datasets can compress teacher GANs at ratios of 1669:1, 58:1, and 87:1, respectively, while retaining the quality of the generated image. From our experiments, we observe a qualitative limit for GAN’s compression. Moreover, we observe that, with a fixed parameter budget, compressed GANs outperform GANs trained using standard training methods. We conjecture that this is partially owing to the optimization landscape of over-parameterized GANs which allows efficient training using alter
中文速览
现有的生成对抗网络（GAN）动辄拥有数千万甚至上亿个参数，根本无法部署到手机等低功耗设备上，但学界此前从未系统研究过如何压缩GAN。本文借鉴知识蒸馏（knowledge distillation）思路，让一个参数量极少的"学生"GAN去模仿一个训练好的大"教师"GAN，通过最小化两者生成图像之间的像素级误差（MSE损失）或同时引入对抗损失的联合目标来完成迁移。在MNIST、CIFAR-10和Celeb-A三个数据集上的实验表明，学生网络可以分别以1669:1、58:1和87:1的压缩比还原教师网络的图像质量；更有意思的是，在参数量相同的前提下，先训练一个过参数化的大GAN再压缩，比直接训练同等规模的小GAN效果更好，原因在于过参数化网络的损失曲面更平坦、更易优化。这项工作首次给出了GAN可压缩性的系统证明和实用方案，为在边缘设备上部署高质量生成模型铺平了道路。
原文 arXiv:1902.00159;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1902.00159v1