Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks
Nils Reimers Iryna Gurevych Ubiquitous Knowledge Processing Lab (UKP) and Research Training Group AIPHES Department of Computer Science, Technische Universität Darmstadt Ubiquitous Knowledge Processing Lab (UKP-DIPF) German Institute for Educational Research www.ukp.tu-darmstadt.de
Abstract
Selecting optimal parameters for a neural network architecture can often make the difference between mediocre and state-of-the-art performance. However, little is published which parameters and design choices should be evaluated or selected making the correct hyperparameter optimization often a “black art that requires expert experiences” [Snoek et al. (2012]. In this paper, we evaluate the importance of different network design choices and hyperparameters for five common linguistic sequence tagging tasks (POS, Chunking, NER, Entity Recognition, and Event Detection). We evaluated over 50.000 different setups and found, that some parameters, like the pre-trained word embeddings or the last layer of the network, have a large impact on the performance, while other parameters, for example the number of LSTM layers or the number of recurrent units, are of minor importance. We give a recommendation on a configuration that performs well among different tasks. The optimized implementation of our BiLSTM-CRF architecture is publicly available.111https://github.com/UKPLab/emnlp2017-bilstm-cnn-crf
中文速览
如何为双向LSTM序列标注模型选对超参数，一直是个靠经验摸索的"玄学"难题。研究者系统评估了词性标注、分块、命名实体识别等五项序列标注任务，在五万余种不同的BiLSTM-CRF网络配置上做了大规模随机搜索实验，逐一衡量每个设计选择对最终性能的实际影响。结果发现，预训练词向量的选择和网络最后一层的类型（是否使用CRF）对性能影响最大，而LSTM层数和每层的隐藏单元数影响相对有限、约100个单元即可；此外，Nadam优化器、梯度归一化、BIO标注方案以及对输出和循环维度同时施加变分dropout等选项也被证实普遍有益。这项工作为实践者提供了一份有实验依据的超参数配置建议，帮助大家把有限的计算资源集中花在真正重要的地方，而不是在影响甚微的参数上反复试错。
原文 arXiv:1707.06799;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1707.06799v2