arXiv：cs.AI· Shihua Yu, Yanzhi Wang, Tianchi Chen·· 16 小时前
递归专家计算的粗粒化：延续相对接口与失效模式
Coarse-Graining Recursive Expert Computation: Continuation-Relative Interfaces and Failure Modes
AI 导读
研究提出延续相对评估框架，以下游任务质量与总执行成本为联合标准衡量计算替换。在 1.7-billion 参数模型上，next-token-loss 训练使分布内负对数似然降低 0.1901 nats/token，但该排名在全部九组分布偏移对比中反转。递归专家原型中联合适配候选优于冻结库与仅路由基线，却仍落后于直接训练的控制组，且测试的仅输入选择器未通过质量-成本联合标准。
正文
Abstract:Computational coarse-graining aims to replace a sequence of updates with a cheaper module while preserving task performance. We present a continuation-relative evaluation framework that assesses replacements through the computation that follows them, using downstream task quality and total execution cost as joint criteria. The framework distinguishes boundary representations, summaries of skipped computation, executed operations, and downstream loss. It makes interfaces and permitted deployment information explicit, combines fine-path references with directly trained controls using the same shortened architecture, and accounts for selection overhead. An exact loss decomposition separates replacement effects at a fixed downstream action from effects of changing that action. A finite construction shows how retained information can reduce the stipulated cost of zero-error prediction under fixed information and controller restrictions. Experiments in two frozen pretrained language models show that representation fidelity and task loss can favor different fitting objectives. In a prespecified comparison on a 1.7-billion-parameter model, next-token-loss training lowers in-distribution negative log-likelihood by 0.1901 nats per scored token compared with reconstructing the skipped blocks' output, using the same optimizer and update count. This ranking reverses in all nine paired comparisons under distribution shift. In a recursive expert prototype, the jointly adapted candidate lowers task error relative to frozen-bank and router-only baselines, but trails directly trained controls, and the tested input-only selectors fail the joint quality-cost criteria. Together, the protocol and studies provide explicit criteria and diagnostics for distinguishing useful computational replacement from local approximation gains within specified interfaces and evaluation settings.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computational Complexity (cs.CC); Computational Engineering, Finance, and Science (cs.CE) |
| Cite as: | arXiv:2610.14413 [cs.LG] |
| (or arXiv:2610.14413v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14413 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tianchi Chen [view email]
[v1]
Sat, 10 Oct 2026 00:21:04 UTC (313 KB)
来源：arXiv：cs.AI · arxiv.org