MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
Longhui Yu1,⋆⋆\star⋆ Weisen Jiang2,3,⋆⋆\star⋆ Han Shi4,† Jincheng Yu3,4 Zhengying Liu4 Yu Zhang2 James T. Kwok3 Zhenguo Li4 Adrian Weller1,5 Weiyang Liu1,6,† 1University of Cambridge 2Southern University of Science and Technology 3Hong Kong University of Science and Technology 4Huawei Noah’s Ark Lab 5The Alan Turing Institute 6Max Planck Institute for Intelligent Systems - Tübingen
Abstract
Large language models (LLMs) have pushed the limits of natural language understanding and exhibited excellent problem-solving ability. Despite the great success, most existing open-source LLMs (e.g., LLaMA-2) are still far away from satisfactory for solving mathematical problems due to the complex reasoning procedures. To bridge this gap, we propose MetaMath, a finetuned language model that specializes in mathematical reasoning. Specifically, we start by bootstrapping mathematical questions by rewriting the question from multiple perspectives, which results in a new dataset called MetaMathQA. Then we finetune the LLaMA-2 models on MetaMathQA. Experimental results on two popular benchmarks (i.e., GSM8K and MATH) for mathematical reasoning demonstrate that MetaMath outperforms a suite of open-source LLMs by a significant margin. Our MetaMath-7B model achieves $66.5\%$ on GSM8K and $19.8\%$ on MATH, exceeding the state-of-the-art models of the same size by $11.5\%$ and $8.7\%$ . Particularly, MetaMath-70B achieves an accuracy of $82.3\%$ on GSM8K, slightly better than GPT-3.5-Turbo. We release the MetaMathQA dataset, the MetaMath models with different model sizes and the training code
中文速览
开源大语言模型在数学推理上表现不佳，根本原因在于训练数据中的题目种类太少、视角太单一。为此，研究者提出了 MetaMath，核心思路是对原始数学题做"题目自举"（question bootstrapping）：用大模型把题目改写成多种表述，同时构造"逆向推理题"——把题目中的某个数字遮住，给出答案让模型反推，从而让模型学会从多个角度理解同一道题背后的数学知识，并以此构建出新数据集 MetaMathQA。基于该数据集微调 LLaMA-2 后，MetaMath-7B 在 GSM8K 和 MATH 两个主流数学基准上分别达到 66.5% 和 19.8%，比此前最好的同规模开源模型高出 11.5 和 8.7 个百分点，而 MetaMath-70B 在 GSM8K 上的 82.3% 甚至略超 GPT-3.5-Turbo。这项工作表明，精心设计的数据增强策略本身就能大幅提升数学推理能力，为开源社区提供了一条无需依赖复杂训练框架就能缩小与闭源模型差距的可行路径。
原文 arXiv:2309.12284;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2309.12284v4