arXiv：cs.LG（机器学习，全量分类）· Qiyu Chen, Guozhang Chen·· 1 天前AI 评分38
为状态空间模型对齐归纳偏置，实现数据高效泛化
Aligning Inductive Bias for Data-Efficient Generalization in State Space Models
AI 导读
研究者提出 Task-Dependent Initialization（TDI），一种快速功率谱匹配方法，可在下游训练前将线性时不变 SSM 的初始归纳偏置与任务的频谱特征对齐。
正文
Abstract:The remarkable success of modern AI has been closely tied to scaling laws, yet the finite supply of high-quality data makes data efficiency--learning more from less--an increasingly important frontier. A model's inductive bias is a critical lever for data efficiency, but foundational sequence models such as State Space Models (SSMs) often rely on fixed, task-agnostic biases. When this fixed prior is misaligned with the underlying structure of a task, the model may require additional samples to overcome its own bias before learning the relevant signal. In this work, we introduce a principled framework for understanding and aligning the inductive bias of linear time-invariant SSMs. We first formalize this bias through an SSM-induced kernel and show theoretically and empirically that its spectrum is governed by the model's frequency response. This characterization motivates Task-Dependent Initialization (TDI), a fast power-spectrum matching method that aligns the initial SSM bias with the task's spectral characteristics before downstream training. Across controlled synthetic experiments, trainable one-layer SSMs, and deep SSMs on diverse real-world benchmarks, TDI can improve data-efficient generalization primarily when task-relevant spectral structure is present and the default SSM bias is spectrally mismatched. Our results provide both a theoretical lens and a practical tool for task-adaptive inductive bias, suggesting a path toward more data-efficient sequence modeling.
| Comments: | NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG) |
| MSC classes: | 68Q32 |
| ACM classes: | I.2.6 |
| Cite as: | arXiv:2509.20789 [cs.LG] |
| (or arXiv:2509.20789v5 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2509.20789 arXiv-issued DOI via DataCite |
Submission history
From: Qiyu Chen [view email]
[v1]
Thu, 25 Sep 2025 06:14:44 UTC (856 KB)
[v2]
Fri, 26 Sep 2025 05:57:47 UTC (856 KB)
[v3]
Thu, 27 Nov 2025 06:46:48 UTC (1 KB) (withdrawn)
[v4]
Tue, 5 May 2026 03:15:54 UTC (2,111 KB)
[v5]
Thu, 1 Oct 2026 03:54:53 UTC (2,131 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org