arXiv：cs.LG· Yijia Jessica Zhu, David Chiang·· 4 小时前AI 评分39
Transformer 的精确解体积与长度泛化研究
Exact-Solution Volume and Length Generalization in Transformers
AI 导读
研究者提出归一化精确解体积（NESV）指标，用于衡量 Transformer 解在更长输入上的可泛化性，并对固定宽度单层 Transformer 在 FIRST、MAJORITY、INDEX、PARITY 四个任务上给出渐近界，分别为 Θ(1)、Θ(1/(n log n))、Θ(1/n³) 和 0。
正文
Abstract:Research on transformer expressivity shows whether a transformer is capable of solving a given task, but gives little indication of whether the solution, if learned, is generalizable to longer input lengths. We study this question through normalized exact-solution volume (NESV): the fraction of a bounded parameter region that achieves an exact solution on every input of length $n$. For fixed-width, single-layer transformers with $\log n$-scaled attention, we establish asymptotic bounds on NESV for four tasks: FIRST ($\Theta(1)$), MAJORITY ($\Theta(1/(n\log n))$), INDEX ($\Theta(1/n^3)$), and PARITY ($0$). These results are consistent with previous empirical results: the faster the exact-solution volume decays with input length, the harder it is to length-generalize on that task. Looking deeper into INDEX, our volume analysis reveals two error sources that grow with $n$. Consequently, we study a transformer model that would structurally eliminate one of the terms, theoretically improving the NESV bound to $\Theta(n^{-1})$, and empirically achieving 85% accuracy when tested at $10\times$ the training length, compared with the 60% accuracy of the original model. We conclude that volume analysis may be a useful approach to identify concrete sources of length sensitivity and thus provide insights into task-specific model refinements.
| Comments: | 26 pages, 2 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.07676 [cs.LG] |
| (or arXiv:2610.07676v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07676 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yijia Jessica Zhu [view email]
[v1]
Tue, 6 Oct 2026 03:11:14 UTC (108 KB)
来源：arXiv：cs.LG · arxiv.org