arXiv：cs.LG· Xianzhi Zeng, Jiangneng Li, Gao Cong·· 5 小时前AI 评分31
超越数据分布：语言模型的系统行为、因果税与非因果基座模型
To Explore The Strange New World Beyond Data Distribution: System Behavior, Causality Tax, and Non-causal Base Model
AI 导读
研究提出 SBD 框架，将系统行为 S 作为证据下界（ELBO）的不可约分量，理论上揭示了"因果税"现象——因果性因忽视 S 而成为带额外结构误差的次优近似。团队构建非因果变分族 Green Shell（GSH），用分治划分替代 S 分量的顺序依赖链，NTK 谱显示其误差界始终显著更紧，信噪比提升 7dB+，懒训练后期多尺度拟合能力提升最多 20%。
正文
Abstract:We show that the causality of language models (LMs) may not be necessary nor optimal. This is the case when system behavior (denoted as $S$) is incorporated as a first-principle Bayesian feature. Here, $S$ refers to extra dominant factors beyond the data space, and they involve coupled effects. Despite being the de facto foundation of modern architecture, recent studies indicate persistent mismatches and contradictions with causality. These issues largely stem from system behavior rather than the data distribution. We therefore propose the SBD framework, which incorporates $S$ as an irreducible component of the evidence lower bound (ELBO). SBD theoretically reveals a counter-intuitive Causality Tax phenomenon, where causality emerges as a suboptimal approximation with an additional structural error, due to the obliviousness to $S$. To address the challenge of latent variable analysis, we validate the SBD-predicted impact of $S$ via implicit measurements, theoretical-bound-guided controls, and Neural Tangent Kernel (NTK) evaluations. In particular, we construct Green Shell (GSH) to show the possibility of reducing Causality Tax. GSH is a non-causal variational family, and it replaces the sequential dependency chain of $S$ components with a divide-and-conquer partition. NTK spectra in the lazy-training regime confirm that GSH always achieves significantly tighter error bounds than causality, with $7dB+$ improvement in signal-to-noise ratio. In the relatively later stage of lazy-training, GSH further leads to superior generalization (up to $20\%$ richer multi-scale fitting capabilities). Taken together, SBD establishes system behavior as a complementary theoretical abstraction besides causality and distribution fitting, opening new research avenues such as designing and optimizing LM base models.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.02839 [cs.LG] |
| (or arXiv:2610.02839v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02839 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiangneng Li [view email]
[v1]
Fri, 2 Oct 2026 05:29:03 UTC (1,764 KB)
来源：arXiv：cs.LG · arxiv.org