arXiv：cs.CL· Raghavv Goel, Sudhanshu Agrawal, Mukul Gagrani, Risheek Garrepalli, Mingu Lee·· 11 小时前
通过激活波分析 LLM 隐藏状态的信息与几何特性以优化投机解码
Surfing the Activation Wave: Analyzing the Information and Geometry of LLM Hidden States for Speculative Decoding
AI 导读
研究提出一套基于信息量与多样性双准则的多层隐藏状态选择框架，用于投机解码。该方法通过矩阵熵衡量信息量、用 CKA 量化多样性，并引入 token 熵过滤剔除低预测熵 token 的隐藏状态。在 EAGLE3 风格的投机解码中，所选层组合优于固定基线（含默认 [2, N//2, N-3] 规则）。
正文
Abstract:Decoder hidden states from different layers of large language models (LLMs) often encode complementary information. However, existing speculative decoding methods typically rely on a small set of target-model layers chosen using fixed, position-based heuristics. Selecting an optimal subset of layers is inherently a combinatorial problem, making exhaustive evaluation infeasible even for moderately deep models. In this work, we present a principled framework for multi-layer hidden-state selection for speculative decoding based on two complementary criteria: informativeness, measured through matrix-entropy analysis, and diversity, quantified using centered kernel alignment (CKA). We investigate layer-wise Shannon entropy, aggregate entropy under hidden-state concatenation, and pairwise CKA dynamics across depth, revealing distinct geometric and informational compartments within decoder representations. Motivated by these findings, we propose entropy and CKA-based layer selection algorithms that favor hidden-state sets that are both highly informative and minimally redundant. As part of this framework, we introduce token-entropy filtering, which removes hidden states associated with low predictive-entropy tokens, thereby improving the quality of layer-selection statistics. In EAGLE3-style speculative decoding, our selected sets outperform fixed baseline choices, including the default $[2, N//2, N-3]$ rule, while providing a clearer explanation of why some defaults can remain competitive. Our results provide a practical recipe for multi-layer hidden-state selection for speculative decoding, insights into the geometry and information in LLM decoder states, and a foundation for extending similar analysis to related applications.
| Comments: | Accepted at multiple Neurips Workshop: Axiom, GDDL, and PriGM |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.12939 [cs.CL] |
| (or arXiv:2610.12939v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12939 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Raghavv Goel Mr. [view email]
[v1]
Fri, 9 Oct 2026 03:38:26 UTC (1,550 KB)
来源：arXiv：cs.CL · arxiv.org