Improving Transformer Models by Reordering their Sublayers
Ofir Press♢ Noah A. Smith♢♠ Omer Levy♣ ♢Paul G. Allen School of Computer Science、Engineering, University of Washington ♠Allen Institute for AI ♣Facebook AI Research
Abstract
Multilayer transformer networks consist of interleaved self-attention and feedforward sublayers. Could ordering the sublayers in a different pattern lead to better performance? We generate randomly ordered transformers and train them with the language modeling objective. We observe that some of these models are able to achieve better performance than the interleaved baseline, and that those successful variants tend to have more self-attention at the bottom and more feedforward sublayers at the top. We propose a new transformer pattern that adheres to this property, the sandwich transformer, and show that it improves perplexity on multiple word-level and character-level language modeling benchmarks, at no cost in parameters, memory, or training time. However, the sandwich reordering pattern does not guarantee performance gains across every task, as we demonstrate on machine translation models. Instead, we suggest that further exploration of task-specific sublayer reorderings is needed in order to unlock additional gains.111Our code is available at https://github.com/ofirpress/sandwich_transformer
中文速览
标准Transformer把自注意力层和前馈层严格交替堆叠，但这种排列方式未必是最优的。研究者通过随机打乱这两类子层的顺序并训练大量模型，发现约三分之一的随机排列能跑赢标准交替结构，而表现更好的模型普遍呈现出"底部自注意力多、顶部前馈层多"的规律。基于这一发现，他们设计了"三明治Transformer"（sandwich transformer）——在网络开头堆几层自注意力、结尾堆几层前馈，中间保留原始交替结构——在不增加任何参数、显存或训练时间的前提下，在多个词级和字符级语言建模基准上都取得了困惑度的明显下降，并在enwik8字符级任务上达到了当时的最优成绩。这项工作表明，仅凭子层的排列顺序就能提升Transformer性能，但也指出该模式在机器翻译上收益有限，暗示不同任务可能需要各自定制的子层排列方式，值得进一步探索。
原文 arXiv:1911.03864;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1911.03864v2