Language Models as Inductive Reasoners
Zonglin Yang♠♠\spadesuit Li Dong♢♢\diamondsuit Xinya Du♣♣\clubsuit Hao Cheng♢♢\diamondsuit Erik Cambria♠♠\spadesuit Xiaodong Liu♢♢\diamondsuit Jianfeng Gao♢♢\diamondsuit Furu Wei♢♢\diamondsuit ♠♠\spadesuit Nanyang Technological University ♢♢\diamondsuit Microsoft Research ♣♣\clubsuit University of Texas at Dallas Contribution during internship at Microsoft Research.
Abstract
Inductive reasoning is a core component of human intelligence. In the past research of inductive reasoning within computer science, formal language is used as representations of knowledge (facts and rules, more specifically). However, formal language can cause systematic problems for inductive reasoning such as disability of handling raw input such as natural language, sensitiveness to mislabeled data, and incapacity to handle ambiguous input. To this end, we propose a new paradigm (task) for inductive reasoning, which is to induce natural language rules from natural language facts, and create a dataset termed DEER containing 1.2k rule-fact pairs for the task, where rules and facts are written in natural language. New automatic metrics are also proposed and analysed for the evaluation of this task. With DEER, we investigate a modern approach for inductive reasoning where we use natural language as representation for knowledge instead of formal language and use pretrained language models as “reasoners”. Moreover, we provide the first and comprehensive analysis of how well pretrained language models can induce natural language rules from natural language facts. We also propose a new
中文速览
人类在归纳推理中天然使用自然语言，但传统计算机归纳推理方法（如归纳逻辑编程，Inductive Logic Programming）依赖形式化符号语言，既无法直接处理自然语言输入，又对标注错误和语义歧义极为敏感。为此，研究团队提出了一种全新范式——直接从自然语言事实中归纳出自然语言规则，并构建了包含1200个规则-事实对的数据集DEER，同时设计了新的自动评估指标。在此基础上，他们以预训练语言模型（Pretrained Language Model, PLM）为推理引擎，借鉴哲学文献中对有效归纳条件的定义，提出了名为CoLM（chain-of-language-models）的多模块框架：一个模型负责生成候选规则，另外四个模型分别从不同维度验证规则质量，从而实现自动筛选。实验结果表明，CoLM在自动评估和人工评估上均显著优于基线方法。这项工作填补了自然语言归纳推理的数据与方法空白，为让机器像人类一样从观察中提炼普遍规律迈出了重要一步。
原文 arXiv:2212.10923;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2212.10923v3