Enhancing the Transformer With Explicit Relational Encoding for Math Problem Solving
Imanol Schlag 1, Paul Smolensky2,3, Roland Fernandez2, Nebojsa Jojic2, Jürgen Schmidhuber1, Jianfeng Gao2 1 The Swiss AI Lab IDSIA / USI / SUPSI, 2 Microsoft Research, Redmond, 3 Johns Hopkins University Work partially done while at Microsoft Research.
Abstract
We incorporate Tensor-Product Representations within the Transformer in order to better support the explicit representation of relation structure. Our Tensor-Product Transformer (TP-Transformer) sets a new state of the art on the recently-introduced Mathematics Dataset containing 56 categories of free-form math word-problems. The essential component of the model is a novel attention mechanism, called TP-Attention, which explicitly encodes the relations between each Transformer cell and the other cells from which values have been retrieved by attention. TP-Attention goes beyond linear combination of retrieved values, strengthening representation-building and resolving ambiguities introduced by multiple layers of standard attention. The TP-Transformer’s attention maps give better insights into how it is capable of solving the Mathematics Dataset’s challenging problems. Pretrained models and code are available online111github.com/ischlag/TP-Transformer.
中文速览
把Transformer的注意力机制用张量积表示（Tensor-Product Representation，TPR）加以扩展，让模型在做注意力时不仅检索"内容"，还同步生成一个"关系向量"并将两者绑定，从而为每个注意力头明确编码出语义关系结构。在包含56类自由形式数学文字题的Mathematics Dataset上，这一张量积Transformer（TP-Transformer）刷新了当时的最优成绩，插值准确率可达84%以上；与标准多头注意力相比，新机制能在多层叠加中规避因线性混合引发的"绑定歧义"问题，使模型在处理需要精确解析和组合计算的数学推理任务时表现更稳定。这项工作的重要性在于它为Transformer引入了一套有理论支撑的结构化表示框架，注意力图也因此变得更易解读，为将符号推理能力融入神经网络提供了新的可行路径。
原文 arXiv:1910.06611;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1910.06611v2