Analysing Mathematical Reasoning Abilities of Neural Models
David Saxton DeepMind、Edward Grefenstette DeepMind、Felix Hill DeepMind、Pushmeet Kohli DeepMind
Abstract
Mathematical reasoning—a core ability within human intelligence—presents some unique challenges as a domain: we do not come to understand and solve mathematical problems primarily on the back of experience and evidence, but on the basis of inferring, learning, and exploiting laws, axioms, and symbol manipulation rules. In this paper, we present a new challenge for the evaluation (and eventually the design) of neural architectures and similar system, developing a task suite of mathematics problems involving sequential questions and answers in a free-form textual input/output format. The structured nature of the mathematics domain, covering arithmetic, algebra, probability and calculus, enables the construction of training and test splits designed to clearly illuminate the capabilities and failure-modes of different architectures, as well as evaluate their ability to compose and relate knowledge and learned processes. Having described the data generation process and its potential future expansions, we conduct a comprehensive analysis of models from two broad classes of the most powerful sequence-to-sequence architectures and find notable differences in their ability to resolve mathem
中文速览
针对深度学习模型在数学推理上能力不足、缺乏系统性评估基准这一痛点，研究者构建了一套覆盖算术、代数、概率和微积分等多个领域的序列到序列（sequence-to-sequence）数学问答数据集，题目以自由文本形式输入和输出，并专门设计了"插值测试"（interpolation test，考查已见难度范围内的泛化）和"外推测试"（extrapolation test，考查超出训练分布的泛化），以区分模型究竟是在记忆还是在真正理解规则。数据集通过程序化生成、先采样答案再反向构造问题的方式，保证题目难度可控、训练集与测试集不重叠，同时支持通过"模块组合"自动生成需要中间推理步骤的复合题目。实验对比了当时最强的几类序列到序列架构，发现它们在部分题型上表现尚可，但整体泛化能力有限，在需要组合推理和知识迁移的题目上明显力不从心。这项工作为衡量神经网络的代数推理与系统泛化能力提供了一个清晰、可扩展的测试平台，也为后续探索机器推理的研究奠定了基础。
原文 arXiv:1904.01557;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1904.01557v1