Draw Me a Flower: Processing and Grounding Abstraction in Natural Language
Royi Lachmy1,2 Valentina Pyatkin1,2 Avshalom Manevich1 Reut Tsarfaty1,2 1Bar-Ilan University, Ramat Gan, Israel 2Allen Institute for Artificial Intelligence, Tel Aviv, Israel {valpyatkin,
Abstract
Abstraction is a core tenet of human cognition and communication. When composing natural language instructions, humans naturally evoke abstraction to convey complex procedures in an efficient and concise way. Yet, interpreting and grounding abstraction expressed in NL has not yet been systematically studied in NLP, with no accepted benchmarks specifically eliciting abstraction in NL. In this work, we set the foundation for a systematic study of processing and grounding abstraction in NLP. First, we deliver a novel abstraction elicitation method and present Hexagons, a 2D instruction-following game. Using Hexagons we collected over 4k naturally-occurring visually-grounded instructions rich with diverse types of abstractions. From these data, we derive an instruction-to-execution task and assess different types of neural models. Our results show that contemporary models and modeling practices are substantially inferior to human performance, and that models’ performance is inversely correlated with the level of abstraction, showing less satisfying performance on higher levels of abstraction. These findings are consistent across models and setups, confirming that abstraction is a chall
中文速览
人类在日常交流中会自然地使用"抽象"来高效表达复杂指令，比如用"每隔一个周三开会"代替逐条列举每次会议，但现有NLP研究从未系统研究过如何让机器理解和执行这类抽象自然语言。研究团队借鉴计算思维（Computational Thinking）教育领域的方法，设计了一款名为Hexagons的二维六边形棋盘协作游戏，通过精心设计的视觉图案（包含对象、循环、条件等规律结构）来隐式诱导玩家在描述绘图步骤时自然产生抽象表达，从而收集了超过4000条包含多种抽象层次的自然语言指令数据集。研究者在此数据集上构建了"指令到执行"任务，并对比了基于规则、分类和生成的多种神经网络模型，结果发现所有模型的表现都远低于人类水平，且抽象层次越高、模型表现越差，这一规律在不同模型和实验设置下高度一致。这项工作首次为NLP领域提供了系统研究自然语言抽象理解的基准数据集和实验框架，揭示了当前AI系统在处理人类日常抽象表达上的显著短板，为未来研究指明了重要方向。
原文 arXiv:2106.14321;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.14321v2