Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling
Peng Zhou1, Zhenyu Qi1, Suncong Zheng1, Jiaming Xu1, Hongyun Bao1, Bo Xu1,2 (1) Institute of Automation, Chinese Academy of Sciences, China (2) Center for Excellence in Brain Science and Intelligence Technology, China {zhoupeng2013, zhenyu.qi, zhengsuncong, jiaming.xu, hongyun.bao, Correspondence author:
Abstract
Recurrent Neural Network (RNN) is one of the most popular architectures used in Natural Language Processsing (NLP) tasks because its recurrent structure is very suitable to process variable-length text. RNN can utilize distributed representations of words by first converting the tokens comprising each text into vectors, which form a matrix. And this matrix includes two dimensions: the time-step dimension and the feature vector dimension. Then most existing models usually utilize one-dimensional (1D) max pooling operation or attention-based operation only on the time-step dimension to obtain a fixed-length vector. However, the features on the feature vector dimension are not mutually independent, and simply applying 1D pooling operation over the time-step dimension independently may destroy the structure of the feature representation. On the other hand, applying two-dimensional (2D) pooling operation over the two dimensions may sample more meaningful features for sequence modeling tasks. To integrate the features on both dimensions of the matrix, this paper explores applying 2D max pooling operation to obtain a fixed-length representation of the text. This paper also utilizes 2D con
中文速览
传统循环神经网络（RNN）处理文本时，会把每个句子转化成一个"时间步×特征向量"的二维矩阵，但现有方法只在时间步这一个维度上做一维池化或注意力操作，完全忽略了特征向量维度上各元素之间的关联，可能丢失重要的结构信息。为此，本文提出将双向长短期记忆网络（BLSTM）输出的矩阵像图像一样对待，分别引入二维最大池化（BLSTM-2DPooling）和二维卷积加二维池化（BLSTM-2DCNN）两种模型，同时在两个维度上提取更有意义的特征。在情感分析、问题分类、主观性分类和新闻组分类共六个文本分类任务上的实验表明，BLSTM-2DCNN在其中四个任务上达到当时最优水平，尤其在斯坦福情感树库（SST）的二分类和五分类任务上均取得最高准确率。这项工作首次将二维卷积和二维池化引入自然语言处理领域，为序列建模提供了一种更充分利用隐状态矩阵结构的新思路。
原文 arXiv:1611.06639;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1611.06639v1