arXiv：cs.AI· Zeyu Fan, Alexandru-Dragos Manolache, Jan van Gemert·· 16 小时前
RNN Homunculus：门控 RNN 如何学习长程记忆
The RNN Homunculus: Learning to Gate Long-Term Memory
AI 导读
门控 RNN（如 GRU、LSTM）的长程记忆能力依赖门值本身的学习，而这一过程同样受梯度消失困扰：当训练样本全部要求长程依赖时，门控架构表现不佳。研究通过加入短依赖样本进行干预，使门控模型学会捕捉长依赖样本并跨类别标签泛化，训练得当的门控 RNN 测试准确率可匹配甚至超过 Mamba 和 xLSTM。
正文
Abstract:Learning long-range dependencies with recurrent neural networks (RNNs) is difficult because of the vanishing gradient problem. Gated recurrent neural networks like GRUs and LSTMs address this problem and are commonly explained by their ability to create additive copy paths through time. This explanation is correct, but incomplete: useful RNN gate values must themselves be learned, which is typically done with another RNN. We study this gate-learning problem through carefully controlled sequence classification tasks. Our analysis shows that gated architectures struggle when all training samples require long-range memory and thus equally suffer from the vanishing gradient problem. We intervene by adding short-dependency samples which let gated models learn solutions that capture long-dependency samples, but also generalize across different class labels. When properly learned, gated RNNs match or even exceed the test accuracy of modern architectures like Mamba and xLSTM. Overall, our results highlight gate learning as a central part of learning long-range memory and show how training data can support this process by exposing the same task over shorter dependencies.
| Comments: | 20 pages, including references and appendices |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14279 [cs.LG] |
| (or arXiv:2610.14279v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14279 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zeyu Fan [view email]
[v1]
Fri, 9 Oct 2026 19:59:35 UTC (2,999 KB)
来源：arXiv：cs.AI · arxiv.org