Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
Gaurav Menghani 0000-0003-2912-2522 Google ResearchMountain ViewCaliforniaUSA95054
Abstract
Deep Learning has revolutionized the fields of computer vision, natural language understanding, speech recognition, information retrieval and more. However, with the progressive improvements in deep learning models, their number of parameters, latency, resources required to train, etc. have all have increased significantly. Consequently, it has become important to pay attention to these footprint metrics of a model as well, not just its quality. We present and motivate the problem of efficiency in deep learning, followed by a thorough survey of the five core areas of model efficiency (spanning modeling techniques, infrastructure, and hardware) and the seminal work there. We also present an experiment-based guide along with code, for practitioners to optimize their model training and deployment. We believe this is the first comprehensive survey in the efficient deep learning space that covers the landscape of model efficiency from modeling techniques to hardware support. Our hope is that this survey would provide the reader with the mental model and the necessary understanding of the field to apply generic efficiency techniques to immediately get significant improvements, and also e
中文速览
深度学习模型在图像识别、自然语言理解等领域屡创佳绩，但代价是模型参数量和计算开销急剧膨胀——GPT-3 拥有 1750 亿参数、训练一次耗资数百万美元——这使得"高效"与"高质量"同等重要。这篇综述系统梳理了高效深度学习（Efficient Deep Learning）的五大核心方向：压缩技术（剪枝、量化等）、学习技术（知识蒸馏、数据增强等）、自动化工具（超参数优化、神经架构搜索）、高效架构设计（卷积层、注意力机制等），以及支撑上述方法落地的基础设施与框架。作者不仅对各方向的代表性工作做了全面梳理，还提供了可直接上手的实验代码指南，帮助从业者在模型训练和部署中快速取得实质性的效率提升。这是目前首篇横跨建模技术到硬件支持、覆盖高效深度学习全貌的综合性综述，对需要在真实业务约束下落地大模型的研究者和工程师极具参考价值。
原文 arXiv:2106.08962;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.08962v2