Understanding by Understanding Not: Modeling Negation in Language Models
Arian Hosseini Mila/Université de Montréal Montréal, Canada、Siva Reddy Mila/McGill University Montréal, Canada、Dzmitry Bahdanau Element AI a ServiceNow Company Montréal, Canada \ANDR Devon Hjelm Mila/Université de Montréal and Microsoft Research Montréal, Canada、Alessandro Sordoni Microsoft Research Montréal, Canada、Aaron Courville Mila/Université de Montréal Montréal, Canada
Abstract
Negation is a core construction in natural language. Despite being very successful on many tasks, state-of-the-art pre-trained language models often handle negation incorrectly. To improve language models in this regard, we propose to augment the language modeling objective with an unlikelihood objective that is based on negated generic sentences from a raw text corpus. By training BERT with the resulting combined objective we reduce the mean top 1 error rate to 4% on the negated LAMA dataset. We also see some improvements on the negated NLI benchmarks.
中文速览
大型预训练语言模型（如BERT）在处理否定句时表现很差——对"古巴的首都不是[MASK]"和"古巴的首都是[MASK]"往往给出相同的答案，说明它根本没有真正理解否定的含义。为此，研究者提出了一套两步走的解决方案：先用句法规则自动把维基百科里的普通句子批量转换成否定句，再引入一种"带参照的非似然训练（unlikelihood training with reference）"——让模型在看到正面陈述作为上下文后，主动降低对应否定句中原答案词的预测概率，同时用知识蒸馏防止模型忘掉原有知识。经过这样的训练，改进后的BERTNOT在否定版LAMA知识填空测试集上的平均Top-1错误率从约9%降至4%，在包含否定的自然语言推理（NLI）测试集上也取得了不同程度的提升，且原有任务的性能基本不受影响。这项工作提供了一种无需人工标注、仅靠自动数据增强就能显著增强模型否定理解能力的轻量级方法，对构建更可靠的语言理解系统具有实际参考价值。
原文 arXiv:2105.03519;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2105.03519v1