Approximation-Aware Dependency Parsing by Belief Propagation
Matthew R. Gormley Mark Dredze Jason Eisner Department of Computer Science Center for Language and Speech Processing Human Language Technology Center of Excellence Johns Hopkins University, Baltimore, MD
Abstract
We show how to train the fast dependency parser of ?) for improved accuracy. This parser can consider higher-order interactions among edges while retaining $O(n^{3})$ runtime. It outputs the parse with maximum expected recall—but for speed, this expectation is taken under a posterior distribution that is constructed only approximately, using loopy belief propagation through structured factors. We show how to adjust the model parameters to compensate for the errors introduced by this approximation, by following the gradient of the actual loss on training data. We find this gradient by back-propagation. That is, we treat the entire parser (approximations and all) as a differentiable circuit, as ?) and ?) did for loopy CRFs. The resulting trained parser obtains higher accuracy with fewer iterations of belief propagation than one trained by conditional log-likelihood.
中文速览
依存句法分析（dependency parsing）中引入高阶特征能显著提升准确率，但随之而来的近似推断（如循环置信传播 loopy belief propagation）会带来系统误差，而传统训练方法对这些误差视而不见。本文的核心思路是：把整个解析流程——包括近似推断、早停、以及最小贝叶斯风险解码——当作一个可微电路，通过反向传播（back-propagation）直接对训练数据上的实际损失求梯度，让模型参数主动补偿近似推断引入的偏差，即所谓"近似感知训练"（ERMA, Empirical Risk Minimization under Approximations）。在 CoNLL-2006/2007 共19种语言以及英语宾州树库的实验中，这种训练方式比标准的条件对数似然（CLL）训练获得了更高的解析准确率，且只需更少轮次的置信传播迭代即可达到更好效果。这项工作的意义在于：它为如何在近似推断广泛存在的结构化预测系统中做到"训练与测试一致"提供了一个清晰可行的范例，对未来将深度学习与结构化推断相结合的模型同样具有方法论价值。
原文 arXiv:1508.02375;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1508.02375v1