Abstract

Reasoning over natural language is a long-standing goal for the research community. However, studies have shown that existing language models are inadequate in reasoning. To address the issue, we present PoEt, a novel reasoning pre-training paradigm. Through pre-training language models with programs and their execution results, PoEt empowers language models to harvest the reasoning knowledge possessed by program executors via a data-driven approach. PoEt is conceptually simple and can be instantiated by different kinds of program executors. In this paper, we showcase two simple instances PoEt-Math and PoEt-Logic, in addition to a complex instance, PoEt-SQL. Experimental results on six benchmarks demonstrate that PoEt can significantly boost model performance in natural language reasoning, such as numerical reasoning, logical reasoning, and multi-hop reasoning. PoEt opens a new gate on reasoning-enhancement pre-training, and we hope our analysis would shed light on the future research of reasoning like program executors.