Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark
Cody Coleman, Daniel Kang∗, Deepak Narayanan∗, Luigi Nardi, Tian Zhao, Jian Zhang, Peter Bailis, Kunle Olukotun, Chris Ré, Matei Zaharia Stanford DAWN Equal Contribution
Abstract
Researchers have proposed hardware, software, and algorithmic optimizations to improve the computational performance of deep learning. While some of these optimizations perform the same operations faster (e.g., increasing GPU clock speed), many others modify the semantics of the training procedure (e.g., reduced precision), and can impact the final model’s accuracy on unseen data. Due to a lack of standard evaluation criteria that considers these trade-offs, it is difficult to directly compare these optimizations. To address this problem, we recently introduced DAWNBench, a benchmark competition focused on end-to-end training time to achieve near-state-of-the-art accuracy on an unseen dataset—a combined metric called time-to-accuracy (TTA). In this work, we analyze the entries from DAWNBench, which received optimized submissions from multiple industrial groups, to investigate the behavior of TTA as a metric as well as trends in the best-performing entries. We show that TTA has a low coefficient of variation and that models optimized for TTA generalize nearly as well as those trained using standard methods. Additionally, even though DAWNBench entries were able to train ImageNet mode
中文速览
深度学习训练越来越烧钱，研究者们提出了大量硬件、软件和算法优化，但这些优化往往在加速计算的同时悄悄损害模型的泛化能力，而现有评估指标（如每轮训练时间）根本看不出这种代价。为此，作者们推出并深入分析了 DAWNBench 基准测试，其核心指标"达到精度所需时间"（Time-to-Accuracy，TTA）将速度与泛化能力绑在一起统一衡量，同时还分析了同样采用 TTA 的 MLPerf v0.5 基准的参赛结果。分析发现：TTA 变异系数普遍低于 14%，是一个相当稳定可靠的指标；针对 TTA 大幅调优的模型，其泛化能力与标准训练方法得到的模型几乎持平，并不存在明显的过拟合风险；不过即便顶尖参赛方已将 ImageNet 训练压缩到 3 分钟以内，硬件利用率仍严重不足——Tensor Core 利用率最低仅有应有水平的十分之一，而分布式训练中通信开销往往超过总时间的一半。这项工作为 ML 训练系统的性能评估提供了严谨的方法论基础，也为未来优化指明了硬件利用率和通信效率两大瓶颈方向。
原文 arXiv:1806.01427;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1806.01427v2