Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks
Shubham Toshniwal1, Sam Wiseman1, Allyson Ettinger2, Karen Livescu1, Kevin Gimpel1 1Toyota Technological Institute at Chicago 2Department of Linguistics, University of Chicago {shtoshni, swiseman, klivescu,
Abstract
Long document coreference resolution remains a challenging task due to the large memory and runtime requirements of current models. Recent work doing incremental coreference resolution using just the global representation of entities shows practical benefits but requires keeping all entities in memory, which can be impractical for long documents. We argue that keeping all entities in memory is unnecessary, and we propose a memory-augmented neural network that tracks only a small bounded number of entities at a time, thus guaranteeing a linear runtime in length of document. We show that (a) the model remains competitive with models with high memory and computational requirements on OntoNotes and LitBank, and (b) the model learns an efficient memory management strategy easily outperforming a rule-based strategy.
中文速览
长篇文档中的指代消解（coreference resolution，即判断文中哪些词语指向同一实体）长期面临内存占用大、计算量随文档长度平方级增长的难题。研究者受人类工作记忆有限这一心理语言学事实启发，发现文档中任意时刻真正"活跃"的实体数量远少于全部实体总数，因此提出了一种只同时追踪少量实体的有界记忆（bounded memory）神经网络模型，当记忆槽满时，模型会自主学习决定"遗忘"哪个旧实体以腾出空间给新实体，从而将运行时间控制在文档长度的线性量级。在文学文本数据集LitBank和新闻数据集OntoNotes上的实验表明，仅用20个记忆槽的有界记忆模型性能与无限制记忆模型相当，同时显著优于基于最近最少使用规则的基线策略，并在LitBank上刷新了最优记录。这项工作为在计算资源受限的场景下处理长文档指代消解提供了切实可行的解决方案。
原文 arXiv:2010.02807;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.02807v3