Pretrained Language Models are Symbolic Mathematics Solvers too!
Kimia Noorbakhsh Sharif University of Technology Modar Sulaiman These authors contributed equally to this work. University of Tartu Mahdi Sharifi∗ University of South Carolina Kallol Roy University of Tartu Pooyan Jamshidi University of South Carolina
Abstract
Solving symbolic mathematics has always been of in the arena of human ingenuity that needs compositional reasoning and recurrence. However, recent studies have shown that large-scale language models such as transformers are universal and surprisingly can be trained as a sequence-to-sequence task to solve complex mathematical equations. These large transformer models need humongous amounts of training data to generalize to unseen symbolic mathematics problems. In this paper, we present a sample efficient way of solving the symbolic tasks by first pretraining the transformer model with language translation and then fine-tuning the pretrained transformer model to solve the downstream task of symbolic mathematics. We achieve comparable accuracy on the integration task with our pretrained model while using around $1.5$ orders of magnitude less number of training samples with respect to the state-of-the-art deep learning for symbolic mathematics. The test accuracy on differential equation tasks is considerably lower comparing with integration as they need higher order recursions that are not present in language translations. We propose the generalizability of our pretrained language mode
中文速览
用预训练语言翻译模型来解符号数学（symbolic mathematics）问题，一直缺乏高效的数据利用方案——现有深度学习方法需要海量训练样本才能泛化。研究者提出先用多语言翻译任务预训练 mBART 等 Transformer 模型，再用少量符号数学数据微调，以此完成函数积分和微分方程的求解。实验表明，该方法在函数积分任务上达到与当前最优深度学习方法相当的精度，所需训练样本却减少了约1.5个数量级；同时在分布偏移场景下表现出更强的鲁棒性，不同语言对预训练的结果也验证了"安娜·卡列尼娜原则"（Anna Karenina Principle）——即在相似环境下训练的模型具有相近的泛化能力。这项工作表明语言翻译与符号数学之间存在可迁移的结构共性，为构建样本高效、计算高效的数学推理系统提供了新思路。
原文 arXiv:2110.03501;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.03501v3