arXiv：cs.AI· Xiahui Zhu, Zhenyi Qi, Wanwei Liu, Haoang Chi, Yang Tan, Xinyu Sun, Ji Wang·· 17 小时前
LLM 自回归解码的鞅刻画：从概率视角分析生成随机动态
Autoregressive Decoding as a Martingale: Characterizing the Stochastic Dynamics of LLM Generation
AI 导读
研究将 LLM 自回归解码的每步预测误差定义为采样 token 指示与模型条件分布之差，证明其累积误差过程构成鞅，条件一阶矩恒为零；二阶可预测二次变差化简为 Σ(1−p(y_t|y_<t))，仅由已实现 token 概率即可计算。在 Llama-3.x 与 Qwen2.5（1B 至 72B）上的白盒实验显示，实测二阶矩与理论预测高度吻合，为 LLM 生成文本检测与不确定性量化提供数学基础。
正文
Abstract:Large language models (LLMs) have achieved remarkable success across a wide range of language tasks. At their core, autoregressive LLMs are conditional probability models that learn to approximate the next-token distribution given the preceding context. Despite this probabilistic nature, the stochastic dynamics of autoregressive decoding have rarely been investigated from a rigorous probabilistic perspective. In this work, we define the per-step prediction error as the difference between the sampled token indicator and the model's conditional predictive distribution, and prove that the cumulative error process is a martingale with respect to the decoding filtration, implying that its conditional first-order moment is identically zero. Beyond this first-order property, we derive a closed-form expression for the second-order behavior of the martingale sequence: its predictable quadratic variation reduces to $\sum_t\left(1-p(y_t\mid y_{<t})\right)$, computable from realized token probabilities alone. This identity establishes a direct theoretical connection between sequence fluctuations and predictive uncertainty during autoregressive decoding. White-box experiments on Llama-3.x and Qwen2.5 families (1B to 72B parameters) verify that the empirically measured second-order moments closely match the theoretical predictions derived from our martingale analysis. These results provide a probabilistic framework for analyzing autoregressive LLM decoding, reveal intrinsic probabilistic fingerprints of different language models, and establish a rigorous mathematical foundation for LLM-generated text detection and uncertainty quantification.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.16041 [cs.LG] |
| (or arXiv:2610.16041v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16041 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiahui Zhu [view email]
[v1]
Mon, 12 Oct 2026 01:31:31 UTC (2,216 KB)
来源：arXiv：cs.AI · arxiv.org