A Tensorized Transformer for Language Modeling
Xindian Ma1, Peng Zhang1 , Shuai Zhang1, Nan Duan2, Yuexian Hou1, Dawei Song3, Ming Zhou2 1College of Intelligence and Computing, Tianjin University, Tianjin, China 2Microsoft Research Asia, Beijing, China 3School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China {xindianma, pzhang, szhang96, {nanduan, Corresponding Author: Peng Zhang
Abstract
Latest development of neural models has connected the encoder and decoder through a self-attention mechanism. In particular, Transformer, which is solely based on self-attention, has led to breakthroughs in Natural Language Processing (NLP) tasks. However, the multi-head attention mechanism, as a key component of Transformer, limits the effective deployment of the model to a resource-limited setting. In this paper, based on the ideas of tensor decomposition and parameters sharing, we propose a novel self-attention model (namely Multi-linear attention) with Block-Term Tensor Decomposition (BTD). We test and verify the proposed attention method on three language modeling tasks (i.e., PTB, WikiText-103 and One-billion) and a neural machine translation task (i.e., WMT-2016 English-German). Multi-linear attention can not only largely compress the model parameters but also obtain performance improvements, compared with a number of language modeling approaches, such as Transformer, Transformer-XL, and Transformer with tensor train decomposition.
中文速览
多头注意力（multi-head attention）机制是 Transformer 的核心组件，但其庞大的参数量让模型难以在资源受限的场景中部署，作者希望在不牺牲性能的前提下大幅压缩这部分参数。他们提出了一种名为"多线性注意力"（Multi-linear Attention）的新型自注意力方法，核心思路是将块项张量分解（Block-Term Tensor Decomposition，BTD）引入注意力层——通过低秩核张量结合查询、键、值三个共享因子矩阵来重建多头机制，同时在多个块之间共享映射参数，从而同时利用低秩近似和参数共享两种压缩策略。在语言建模（PTB、WikiText-103、One-Billion）和英德机器翻译（WMT-2016）四项任务上的实验表明，该方法不仅能以更高的压缩比大幅减少参数量，还在多项指标上超越了原始 Transformer、Transformer-XL 等基线模型。这项工作为在边缘设备或低资源环境中部署大型预训练语言模型提供了一条切实可行的路径。
原文 arXiv:1906.09777;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1906.09777v3