Transformers without Tears: Improving the Normalization of Self-Attention
Toan Q. Nguyen University of Notre Dame、Julian Salazar Amazon AWS AI Equal contribution. Work done during an internship at Amazon AWS AI.
Abstract
We evaluate three simple, normalization-centric changes to improve Transformer training. First, we show that pre-norm residual connections (PreNorm) and smaller initializations enable warmup-free, validation-based training with large learning rates. Second, we propose $\ell_{2}$ normalization with a single scale parameter (ScaleNorm) for faster training and better performance. Finally, we reaffirm the effectiveness of normalizing word embeddings to a fixed length (FixNorm). On five low-resource translation pairs from TED Talks-based corpora, these changes always converge, giving an average +1.1 BLEU over state-of-the-art bilingual baselines and a new 32.8 BLEU on IWSLT '15 English-Vietnamese. We observe sharper performance curves, more consistent gradient norms, and a linear relationship between activation scaling and decoder depth. Surprisingly, in the high-resource setting (WMT '14 English-German), ScaleNorm and FixNorm remain competitive but PreNorm degrades performance. Preprocessing scripts and code are released at https://github.com/tnq177/transformers_without_tears.
中文速览
传统Transformer在训练时依赖复杂的学习率预热（warmup）调度且容易不收敛，尤其在低资源机器翻译场景下问题更突出。研究者提出三项以归一化（normalization）为核心的改进：将层归一化移到残差子层之前（PreNorm）、用单一可学习缩放参数的ℓ₂归一化替换原有LayerNorm（ScaleNorm），以及将词向量固定为统一长度（FixNorm）。在五个低资源TED演讲翻译任务上，这三项改动叠加后平均提升1.1 BLEU，并在IWSLT '15英越翻译上刷新了32.8 BLEU的最优成绩，同时训练更稳定、速度更快，甚至无需预热也能正常收敛。这项工作证明，仅凭对归一化策略的精细调整就能在不增加模型复杂度的前提下显著提升低资源翻译性能，为后续轻量级Transformer优化提供了清晰的实践指引。
原文 arXiv:1910.05895;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1910.05895v2