A Comparative Study on Regularization Strategies for Embedding-based Neural Networks
Hao Peng, 1 Lili Mou,∗1 Ge Li,†1 Yunchuan Chen,2 Yangyang Lu,1 Zhi Jin1 1Software Institute, Peking University, 100871, P. R. China {penghao.pku, luyy11, 2University of Chinese Academy of Sciences, Equal contribution. †Corresponding author.
Abstract
This paper aims to compare different regularization strategies to address a common phenomenon, severe overfitting, in embedding-based neural networks for NLP. We chose two widely studied neural models and tasks as our testbed. We tried several frequently applied or newly proposed regularization strategies, including penalizing weights (embeddings excluded), penalizing embeddings, re-embedding words, and dropout. We also emphasized on incremental hyperparameter tuning, and combining different regularizations. The results provide a picture on tuning hyperparameters for neural NLP models.
中文速览
针对NLP领域中基于词向量（word embedding）的神经网络普遍存在的严重过拟合问题，研究者在关系抽取和情感分析两个任务上系统比较了四种正则化策略：对权重施加L2惩罚、对词向量施加L2惩罚、词向量再嵌入（re-embedding）以及Dropout。实验发现，L2惩罚对词向量的约束不仅能缓解过拟合，还能意外地提升训练精度，而再嵌入方法基本无效；此外，正则化系数可以在训练过程中渐进式调整（即先训练后加正则），效果与一开始就加正则相近，大幅降低了调参成本；将权重L2惩罚与Dropout联合使用能进一步提升泛化性能，两者在一定范围内互补。这项研究为NLP从业者在实际调参时应选择哪种正则化手段、如何组合使用提供了系统的实验参考。
原文 arXiv:1508.03721;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1508.03721v1