arXiv：cs.LG（机器学习，全量分类）· Beatriz Almeida Felicio·· 13 小时前AI 评分38
压缩语言模型中散度如何转化为决策翻转
How Divergence Becomes Decision Flips in Compressed Language Models
AI 导读
研究发现，衡量压缩语言模型决策变化应看总变差而非 KL 散度：在 19 个开源模型的 802 个压缩与扰动副本上，arg-max token 翻转率与总变差之比中位数为 1.05，无需拟合常数。
正文
Abstract:Compression reports summarize how far a compressed language model moved from the dense one, usually by a KL divergence; a deployment that relies on the dense model's outputs needs to know how many of its decisions changed. We show that total variation, not KL, answers this directly. Across 802 compressed and perturbed copies of 19 open models on five corpora and nine mechanically unrelated perturbation families, the rate at which the arg-max token changes (the \emph{flip rate}) tracks total variation at a ratio with median $1.05$, with no fitted constant. KL converts into flips only through its square root and a factor that varies fourfold across models and corpora, because KL averages over tokens before the root is taken; first-order statistics averaged per token, such as Hellinger distance, avoid this, but reports rarely give them. As a result, of two compressors reported on different models and corpora whose flip rates differ by at least $10%$, KL assigns the smaller divergence to the one that changes more decisions in $11%$ of cases, total variation in $1%$. Two pre-registered tests mark the limits: on a held-out code corpus the ratio held for all eight models while three predictions about KL each failed for half of them or more, and on three new models with real kernels it stayed in its band for 37 of 38 checkpoints but fell below one on code for two models. In vLLM speculative decoding, total variation measured under teacher forcing predicts greedy draft acceptance with a mean relative error of $1.1$--$2.4%$, without the task-specific calibration that KL needs.
| Comments: | Preprint |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.00694 [cs.CL] |
| (or arXiv:2610.00694v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.00694 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Beatriz Felicio [view email]
[v1]
Wed, 30 Sep 2026 20:37:18 UTC (185 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org