MathBERT: A Pre-Trained Model for Mathematical Formula Understanding
Shuai Peng, Ke Yuan, Liangcai Gao, Zhi Tang Peking University
Abstract
Large-scale pre-trained models like BERT, have obtained a great success in various Natural Language Processing (NLP) tasks, while it is still a challenge to adapt them to the math-related tasks. Current pre-trained models neglect the structural features and the semantic correspondence between formula and its context. To address these issues, we propose a novel pre-trained model, namely MathBERT, which is jointly trained with mathematical formulas and their corresponding contexts. In addition, in order to further capture the semantic-level structural features of formulas, a new pre-training task is designed to predict the masked formula substructures extracted from the Operator Tree (OPT), which is the semantic structural representation of formulas. We conduct various experiments on three downstream tasks to evaluate the performance of MathBERT, including mathematical information retrieval, formula topic classification and formula headline generation. Experimental results demonstrate that MathBERT significantly outperforms existing methods on all those three tasks. Moreover, we qualitatively show that this pre-trained model effectively captures the semantic-level structural informat
中文速览
数学公式理解长期面临两大难题：公式本身的树形结构复杂，且公式与周围文本之间的语义关联难以被通用预训练模型捕捉。为此，研究团队提出 MathBERT，以 BERT 为骨架，将公式的 LaTeX 符号序列、上下文文本和算符树（Operator Tree, OPT）三路拼接后联合输入，并设计了三个预训练任务：沿用掩码语言建模（MLM）学习文本表示、用上下文对应预测（CCP）建立公式与上下文的语义关联、以及全新的掩码子结构预测（MSP）让模型学会从树结构中恢复被遮盖节点的父子关系。在从 arXiv 抽取的 870 万公式-上下文对上完成预训练后，MathBERT 在数学信息检索、公式主题分类和公式标题生成三个下游任务上均显著超越现有方法。这是首个专为数学公式理解设计的预训练模型，为后续数学智能研究提供了通用的基础表示工具。
原文 arXiv:2105.00377;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2105.00377v1