arXiv：cs.LG· Yupeng Yao·· 4 小时前AI 评分36
轨迹特征能否提升末层注意力路由？SmolLM3-3B-Base 与 Qwen3.5-4B-Base 上的评估
Evaluating Trajectory Features for Routing Final-Layer Attention
AI 导读
一项研究在冻结的 SmolLM3-3B-Base 和 Qwen3.5-4B-Base 检查点上评估隐状态外推误差、曲率与误差变化能否比不确定性、单步位移等信号更好地预测注意力路由价值，在 100 本 PG-19 留出书籍、20% 因果调用配额下，六项预设比较经族系校正后均无正向增益。
正文
Abstract:Attention routing requires a signal that predicts the value of attention on the current prefix. We evaluate whether hidden-state extrapolation error, curvature and error change improve this prediction beyond uncertainty, one-step displacement, position and state projections. Paired executions of the final attention layer supply signed next-token loss differences in frozen SmolLM3-3B-Base and Qwen3.5-4B-Base checkpoints. Utility-supervised routers are tested on 100 held-out PG-19 books at an identical causal 20 percent invocation quota. None of six prespecified comparisons shows a positive gain after familywise correction. In Qwen3.5, a parameter-matched fixed-projection control lowers NLL by 0.00356 nats/token relative to the trajectory router (95 percent interval 0.00218 to 0.00487). Secondary results depend on the operation removed, feature location and scoring horizon; frozen thresholds also drift substantially at longer horizons. Actual selected-query execution yields small long-sequence latency reductions with increased NLL, while learned routers remain slower during cached continuation. The study identifies limits on the incremental value of these trajectory summaries and separates allocation quality from measured inference benefit.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.09272 [cs.LG] |
| (or arXiv:2610.09272v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.09272 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yupeng Yao [view email]
[v1]
Wed, 7 Oct 2026 01:12:37 UTC (248 KB)
来源：arXiv：cs.LG · arxiv.org