Dual-View Distilled BERT for Sentence Embedding
Xingyi Cheng Ant Group
Abstract
Recently, BERT realized significant progress for sentence matching via word-level cross sentence attention. However, the performance significantly drops when using siamese BERT-networks to derive two sentence embeddings, which fall short in capturing the global semantic since the word-level attention between two sentences is absent. In this paper, we propose a Dual-view distilled BERT (DvBERT) for sentence matching with sentence embeddings. Our method deals with a sentence pair from two distinct views, i.e., Siamese View and Interaction View. Siamese View is the backbone where we generate sentence embeddings. Interaction View integrates the cross sentence interaction as multiple teachers to boost the representation ability of sentence embeddings. Experiments on six STS tasks show that our method outperforms the state-of-the-art sentence embedding methods significantly.
中文速览
句子语义匹配任务中，把两句话分别编码成向量再做相似度计算（孪生网络方案）虽然推理效率高，却因为缺少句间词级交互而损失了大量语义信息，性能明显落后于让两句话直接拼接后交互的方案。为此，作者提出 DvBERT（双视角蒸馏 BERT），同时从"孪生视角"和"交互视角"处理句对：孪生视角负责生成句子向量，交互视角则用多个带跨句注意力的预训练模型充当教师，通过知识蒸馏（knowledge distillation）将词级交互信息"灌入"孪生网络的句子表示，并配合教师退火策略让学生网络先学软标签再逐渐转向硬标签。在六个语义文本相似度（STS）基准上的实验表明，DvBERT 相比 SBERT 等孪生网络方案平均提升约 0.56%–1.9% Spearman 相关系数，同时保留了独立编码句子、推理复杂度为 O(n) 的高效检索能力，为在效率与精度之间取得更好平衡提供了切实可行的思路。
原文 arXiv:2104.08675;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2104.08675v1