A Comparative Study of Transformer-Based Language Models on Extractive Question Answering
Kate Pearce1, Tiffany Zhan1, Aneesh Komanduri2, Justin Zhan2 2Department of Computer Science and Computer Engineering, University of Arkansas 1 Army Educational Outreach Program UNITE
Abstract
Question Answering (QA) is a task in natural language processing that has seen considerable growth after the advent of transformers. There has been a surge in QA datasets that have been proposed to challenge natural language processing models to improve human and existing model performance. Many pre-trained language models have proven to be incredibly effective at the task of extractive question answering. However, generalizability remains as a challenge for the majority of these models. That is, some datasets require models to reason more than others. In this paper, we train various pre-trained language models and fine-tune them on multiple question answering datasets of varying levels of difficulty to determine which of the models are capable of generalizing the most comprehensively across different datasets. Further, we propose a new architecture, BERT-BiLSTM, and compare it with other language models to determine if adding more bidirectionality can improve model performance. Using the F1-score as our metric, we find that the RoBERTa and BART pre-trained models perform the best across all datasets and that our BERT-BiLSTM model outperforms the baseline BERT model.
中文速览
让模型真正"读懂"文章而不只是"找词"，是问答系统（Question Answering, QA）领域的核心难题——现有模型在一个数据集上表现出色，换一个需要更多推理的数据集就会大幅退步。研究者在 NewsQA、SQuAD 2.0、QuAC 等难度各异的数据集上，系统比较了 BERT、ALBERT、XLNet、RoBERTa、ConvBERT、BART 等多个预训练语言模型的泛化能力，同时提出了一种新架构 BERT-BiLSTM——先用 BERT 做深层语义编码，再接一层双向长短期记忆网络（BiLSTM）进一步捕捉上下文信息，最后通过线性层预测答案区间。实验结果显示，RoBERTa 和 BART 在所有数据集上综合表现最佳，而 BERT-BiLSTM 则稳定超越了单独使用 BERT 的基线模型。这项工作为如何选择和改进预训练模型以应对不同难度的问答任务提供了系统性参考，也验证了在 Transformer 编码之后叠加双向循环结构是一条可行的性能提升路径。
原文 arXiv:2110.03142;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.03142v1