A3: Accelerating Attention Mechanisms in Neural Networks with Approximation
Tae Jun Ham* Sung Jun Jung* Seonghak Kim* Young H. Oh† Yeonhong Park* Yoonho Song* Jung-Hun Park* Sanghee Lee* Kyoung Park‡ Jae W. Lee* Deog-Kyoon Jeong* *Seoul National University †Sungkyunkwan University ‡SK Hynix {taejunham, miguel92, ksh1102, ilil96, yhsong, jhpark94, shlee95, jaewlee,
Abstract
With the increasing computational demands of neural networks, many hardware accelerators for the neural networks have been proposed. Such existing neural network accelerators often focus on popular neural network types such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs); however, not much attention has been paid to attention mechanisms, an emerging neural network primitive that enables neural networks to retrieve most relevant information from a knowledge-base, external memory, or past states. The attention mechanism is widely adopted by many state-of-the-art neural networks for computer vision, natural language processing, and machine translation, and accounts for a large portion of total execution time. We observe today’s practice of implementing this mechanism using matrix-vector multiplication is suboptimal as the attention mechanism is semantically a content-based search where a large portion of computations ends up not being used. Based on this observation, we design and architect $A^{3}$ , which accelerates attention mechanisms in neural networks with algorithmic approximation and hardware specialization. Our proposed accelerator achieves multip
中文速览
注意力机制（attention mechanism）在计算机视觉、自然语言处理等领域的神经网络中被广泛使用，却占据了大量运行时间和能耗，现有加速器对此支持不足。研究者发现，注意力机制本质上是一种基于内容的相似度搜索，而非真正的稠密矩阵运算，绝大多数计算结果在经过 softmax 归一化后几乎为零、对最终输出毫无贡献，因此存在大量可剪除的冗余计算。基于这一洞察，他们提出了硬件加速器 A³，通过算法层面的近似候选筛选（先快速找出最可能相关的少数条目，跳过无用的全量搜索）与专用硬件流水线（针对注意力运算定制数据通路并充分挖掘并行性）的协同设计，大幅削减实际计算量。实验结果表明，A³ 相比传统 CPU/GPU 硬件实现了多个数量级的能效提升和显著加速，同时对模型精度的影响极小，为在更大知识库或更长序列上部署注意力机制提供了切实可行的硬件基础。
原文 arXiv:2002.10941;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2002.10941v1