arXiv：cs.LG· Aleix Salvador-Pomarol, Arthur N. Montanari, Earl K. Miller, Adilson E. Motter, Jorge Cort\'es·· 4 小时前
二阶递归模型通过空间神经计算学习无限上下文窗口
Learning infinite context windows in recurrent architectures via spatial neural computing
AI 导读
研究提出一种二阶递归模型，用离散化偏微分方程驱动的空间演化场取代传统神经元间通信，构建出等效的结构化无限阶 RNN，当前状态显式依赖全部历史状态，在固定参数量下获得有效无界感受野。该模型通过将梯度谱约束在单位圆上保证边际稳定性，消除梯度消失与爆炸，在长程基准上以显著更少的参数超越其他递归模型。
正文
Abstract:Recurrent neural networks (RNNs) offer linear-time scaling with sequence length while requiring only constant memory, yet they struggle to capture long-range dependencies due to vanishing gradients and limited receptive fields. To address these limitations, we introduce a second-order recurrent model in which the standard neuron-to-neuron communication is replaced by a spatially evolving field governed by (discretized) partial differential equations. Drawing inspiration from the role of cortical waves in brain computation, this mechanism allows structured spatiotemporal patterns to serve as an implicit, high-capacity memory. We show that the resulting model is equivalent to a structured infinite-order RNN in which the current state depends explicitly on its entire history of past states, yielding an effectively unbounded receptive field with a fixed number of parameters. We further derive constructive conditions to ensure marginal stability, constraining the gradient spectrum on the unit circle and thereby eliminating vanishing and exploding gradients. Empirically, the proposed architecture outperforms other recurrent models on long-horizon benchmarks while using substantially fewer parameters, demonstrating that spatial dynamics can effectively bridge the gap between efficient inference and long-term memory.
| Comments: | 12 pages, 4 figures |
| Subjects: | Machine Learning (cs.LG); Systems and Control (eess.SY); Neurons and Cognition (q-bio.NC) |
| Cite as: | arXiv:2610.10690 [cs.LG] |
| (or arXiv:2610.10690v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10690 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Aleix Salvador Pomarol [view email]
[v1]
Wed, 7 Oct 2026 18:00:02 UTC (2,111 KB)
来源：arXiv：cs.LG · arxiv.org