LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning
Yuhuai Wu Markus Rabe Wenda Li Jimmy Ba Roger Grosse Christian Szegedy
Abstract
While designing inductive bias in neural architectures has been widely studied, we hypothesize that transformer networks are flexible enough to learn inductive bias from suitable generic tasks. Here, we replace architecture engineering by encoding inductive bias in the form of datasets. Inspired by Peirce’s view that deduction, induction, and abduction are the primitives of reasoning, we design three synthetic tasks that are intended to require the model to have these three abilities. We specifically design these tasks to be synthetic and devoid of mathematical knowledge to ensure that only the fundamental reasoning biases can be learned from these tasks. This defines a new pre-training methodology called “LIME” (Learning Inductive bias for Mathematical rEasoning). Models trained with LIME significantly outperform vanilla transformers on four very different large mathematical reasoning benchmarks. Unlike dominating the computation cost as traditional pre-training approaches, LIME requires only a small fraction of the computation cost of the typical downstream task. The code for generating LIME tasks is available at https://github.com/tonywu95/LIME.
中文速览
数学推理是人工智能的难题，现有方法通常靠设计复杂网络结构或用海量自然语言语料做预训练来提升模型能力，前者难以覆盖抽象推理需求，后者计算代价极高。受逻辑学家皮尔斯"演绎、归纳、溯因是推理三原语"的启发，研究者设计了三类完全合成、不含任何数学知识的字符串替换任务，分别对应这三种推理能力，用它们对Transformer做轻量级预训练，这一方法被命名为LIME（Learning Inductive bias for Mathematical rEasoning）。实验表明，LIME预训练模型在IsarStep、HOList、MetaMathStep、LeanStep四个数学推理基准上均大幅超越未预训练的基线，其中LeanStep的top-1准确率从15.5%跃升至29.8%，而整个预训练过程在单块GPU上仅需约两小时，计算量远低于下游任务本身。这项工作揭示了预训练的另一面——即便没有任何内容知识，仅靠合理设计的合成任务就能向模型注入有益的推理归纳偏置（inductive bias），为在数据稀缺的数学形式化领域提升模型推理能力提供了一条高效新路径。
原文 arXiv:2101.06223;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2101.06223v2