Why and when should you pool? Analyzing Pooling in Recurrent Architectures
Pratyush Maini†, Keshav Kolluru†, Danish Pruthi‡, Mausam† †Indian Institute of Technology, Delhi, India ‡Carnegie Mellon University, Pittsburgh, USA {pratyush.maini,
Abstract
Pooling-based recurrent neural architectures consistently outperform their counterparts without pooling on sequence classification tasks. However, the reasons for their enhanced performance are largely unexamined. In this work, we explore three commonly used pooling techniques (mean-pooling, max-pooling, and attention111Attention aggregates representations via a weighted sum, thus we consider it under the umbrella of pooling in this paper.), and propose max-attention, a novel variant that captures interactions among predictive tokens in a sentence. Using novel experiments, we demonstrate that pooling architectures substantially differ from their non-pooling equivalents in their learning ability and positional biases: (i) pooling facilitates better gradient flow than BiLSTMs in initial training epochs, and (ii) BiLSTMs are biased towards tokens at the beginning and end of the input, whereas pooling alleviates this bias. Consequently, we find that pooling yields large gains in low resource scenarios, and instances when salient words lie towards the middle of the input. Across several text classification tasks, we find max-attention to frequently outperform other pooling techniques.22
中文速览
在文本分类任务里，带池化（pooling）的循环神经网络（RNN）普遍比不带池化的表现更好，但背后的原因一直没被认真研究过。这篇论文系统分析了均值池化、最大值池化和注意力机制三种常见池化方式在双向长短期记忆网络（BiLSTM）上的工作原理，发现它们的核心优势体现在两点：一是在训练初期为中间位置的隐藏状态提供更直接的梯度通路，避免了梯度消失导致模型过早只依赖句子首尾词语来做分类；二是消除了BiLSTM天然对句首和句尾词汇更敏感的位置偏差，让模型能公平地关注句子中间的关键词。基于这两点认识，研究者还提出了一种新方法——最大注意力（max-attention），用最大值池化的结果作为注意力机制的查询向量，从而捕捉句中重要词语之间的交互关系，在多个分类任务上频繁超越其他池化方式。这项工作不仅解释了"为什么池化有效"，还明确指出它在低资源场景和关键词位于句子中部的情况下收益最大，为实践中选择和设计RNN架构提供了清晰的指导依据。
原文 arXiv:2005.00159;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2005.00159v2