arXiv：cs.AI· Swathi Shree Narashiman, Aravind Ramana Venkatramanan, Nambi Seshadri·· 5 小时前
语言模型 Logits 的几何结构：通用模式与习得性偏离
The Geometry of Language Model Logits: Generic Structure and Learned Departures
AI 导读
研究首次系统分析自回归语言模型 logits 的几何结构，发现有序词表头部 logits 在不同预测上下文、架构和规模下均遵循 log-linear 关系 z_(r) ≈ μ - α log r，并在预训练中逐步显现。
正文
Abstract:Autoregressive language models produce, at every prediction step, a vocabulary-sized logit vector whose structure determines the conditional next-token distribution. We empirically identify that the ordered vocabulary-head logits follow a remarkably consistent log-linear relation
\[ z_{(r)} \approx \mu - \alpha \log r. \] across prediction contexts, architectures, and scales, emerging progressively during pretraining. To our knowledge, this is the first systematic analysis of the logit geometry underlying the distributional assumptions used in the prior decoding work. While controlled order-statistic baselines spanning bounded, light-tailed, and heavier-tailed distributions can largely explain the observed log-linear geometry, they do not explain the systematic shift in the slope $\alpha$: trained language models exhibit $\alpha \approx 1.4$--$1.7$, sitting well above the initialization baseline ($\alpha \approx 0.2$) . Moreover, $\alpha$ correlates with predictive performance, suggesting that it captures learned structure beyond generic extreme-value effects.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.18520 [cs.CL] |
| (or arXiv:2610.18520v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18520 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Swathi Shree Narashiman [view email]
[v1]
Tue, 13 Oct 2026 15:11:06 UTC (653 KB)
来源：arXiv：cs.AI · arxiv.org