Extensions and Limitations of the Neural GPU
Eric Price The University of Texas at Austin、Wojciech Zaremba, Ilya Sutskever OpenAI Work done while at OpenAI.
Abstract
The Neural GPU is a recent model that can learn algorithms such as multi-digit binary addition and binary multiplication in a way that generalizes to inputs of arbitrary length. We show that there are two simple ways of improving the performance of the Neural GPU: by carefully designing a curriculum, and by increasing model size. The latter requires a memory efficient implementation, as a naive implementation of the Neural GPU is memory intensive. We find that these techniques increase the set of algorithmic problems that can be solved by the Neural GPU: we have been able to learn to perform all the arithmetic operations (and generalize to arbitrarily long numbers) when the arguments are given in the decimal representation (which, surprisingly, has not been possible before). We have also been able to train the Neural GPU to evaluate long arithmetic expressions with multiple operands that require respecting the precedence order of the operands, although these have succeeded only in their binary representation, and not with perfect accuracy.
中文速览
神经图形处理单元（Neural GPU）本可以学习二进制加法、乘法等算法并推广到任意长度的输入，但在十进制算术和更复杂的多操作数表达式上一直表现不佳。研究团队通过两项改进——精心设计的课程学习（curriculum learning）策略和更大的模型规模（配合内存优化实现）——成功让 Neural GPU 首次学会了十进制四则运算并能推广到任意位数，还初步实现了需要遵守运算优先级的多操作数算术表达式的求值。然而实验也揭示了一个令人担忧的失败模式：即便模型在随机抽取的长数字测试上准确率接近完美，它仍会在高度对称的特殊输入（如 000…002 × 000…002）上出错，而对普通的 2×2 却毫无问题——这与对抗样本（adversarial examples）的现象高度相似。这表明 Neural GPU 并未真正习得底层算法的通用逻辑，如何让神经网络实现真正意义上的算法泛化仍是一个开放问题。
原文 arXiv:1611.00736;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1611.00736v2