Surprisal-Driven Feedback in Recurrent Networks
Kamil Rocki IBM Research San Jose, CA 95120, USA
Abstract
Recurrent neural nets are widely used for predicting temporal data. Their inherent deep feedforward structure allows learning complex sequential patterns. It is believed that top-down feedback might be an important missing ingredient which in theory could help disambiguate similar patterns depending on broader context. In this paper, we introduce surprisal-driven recurrent networks, which take into account past error information when making new predictions. This is achieved by continuously monitoring the discrepancy between most recent predictions and the actual observations. Furthermore, we show that it outperforms other stochastic and fully deterministic approaches on enwik8 character level prediction task achieving 1.37 BPC.
中文速览
循环神经网络擅长处理序列数据，但缺乏"自上而下的反馈"机制，导致它在需要结合更广泛上下文来消歧的场景中表现有限。这篇论文提出了一种"惊异驱动循环网络"（surprisal-driven recurrent network），核心思路是：在每一步预测时，把上一步预测值与真实观测值之间的差异（即"惊异值"，surprisal）作为额外的输入信号反馈给网络，让模型在推理阶段也能利用自身的预测误差来修正后续判断。研究者将这一机制集成到长短期记忆网络（LSTM）中，通过在隐藏层新增一个反馈连接矩阵来实现，并在经典的enwik8字符级文本预测基准上进行了验证。结果显示，该方法在没有加入任何正则化手段的情况下，以1.37 BPC的成绩超越了当时所有随机和确定性方法，说明把预测误差信息引入推理过程是一种简单而有效的改进思路，也为理解大脑中类似的反馈机制提供了计算层面的启示。
原文 arXiv:1608.06027;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1608.06027v4