arXiv：cs.LG· Shivesh Rane·· 11 小时前
ResonatorLM 的输入依赖读出相位：让共振序列模型在训练长度外不再失效
An Input-Dependent Readout Phase for Resonant Sequence Models
AI 导读
ResonatorLM 将每个 head 固定的读出相位改为当前 token 的函数，只让读出依赖输入，保持精确卷积，仅增加 0.2% 参数。在 WikiText-2 上的 6M 参数字符级模型中，以 256 token 训练、1024 token 测试时，困惑度从 12.81 降至 3.76。
正文
Abstract:Sequence models with a fixed-size state avoid the quadratic cost of attention, but face a design choice. An input-dependent state transition gives content-dependent memory at the cost of FFT-based training, while a fixed transition keeps the O(T log T) convolutional path but reads memory the same way for every input. ResonatorLM, a recent model of the second kind, reads a bank of damped complex oscillators through a fixed learned phase per head. We make that readout phase a function of the current token, which adds input dependence to the read only, keeps the convolution exact, and costs 0.2 percent more parameters. An analysis of the untrained bank shows that phase does encode elapsed time, but an ordinary token falls below the background within 2-8 steps and no choice of damping extends linear reconstruction beyond about 16 tokens. In 6M-parameter character-level models on WikiText-2, the input-dependent phase matches the original at the training length and removes its failure beyond it: trained on 256 tokens and tested on 1024, perplexity falls from 12.81 to 3.76. We trace the failure to long-memory heads that accumulate more history than training exposed them to. The learned phase tracks the current character rather than distance and does not improve retrieval.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.12496 [cs.LG] |
| (or arXiv:2610.12496v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12496 arXiv-issued DOI via DataCite |
Submission history
From: Shivesh Rane [view email]
[v1]
Sat, 26 Sep 2026 15:11:18 UTC (969 KB)
来源：arXiv：cs.LG · arxiv.org