arXiv：cs.LG· Jia-Shu Pan, Tao Zhang, Yufei Huang, Yanjun Sheng, Tailin Wu·· 7 小时前AI 评分46
扩散模型中的特征信息动力学
Feature Information Dynamics in Diffusion
AI 导读
研究者提出"特征信息动力学"这一信息论框架，用于定位扩散模型中特征在何时被生成。该框架借助 I-MMSE 恒等式，将特征互信息变化率与最优无条件去噪损失和特征条件去噪损失之间的差距联系起来，并给出特征信息密度的实用估计方法。
正文
Abstract:Diffusion models generate data through a continuum of denoising problems, and are widely observed to reveal coarse structure before fine detail. Yet, this intuition is mostly empirical and qualitative. We introduce feature information dynamics, an information-theoretic framework for localizing when a feature is generated during diffusion. Using the I-MMSE identity, we connect the rate of feature mutual information change to a gap between optimal unconditional and feature-conditional denoising losses, yielding practical estimators for feature information density. We further develop a chained decomposition that separates shared from incremental information in a feature hierarchy. We use this framework first to quantitatively confirm spectral autoregression in pixel diffusion, and then to extend the analysis beyond frequency: under a class $\to$ mask $\to$ Canny conditioning chain, the per-feature information densities differ across pixel, SDVAE, VAVAE, and RAE, exposing fundamental differences between these representations and suggesting that ordered generation could be beneficial for training diffusion models. Our code is available at this https URL.
| Comments: | Accepted as poster at NeurIPS 2026. 28 pages, including references, appendices, and checklist |
| Subjects: | Machine Learning (stat.ML); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.08626 [stat.ML] |
| (or arXiv:2610.08626v1 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2610.08626 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiashu Pan [view email]
[v1]
Tue, 6 Oct 2026 16:24:07 UTC (1,012 KB)
来源：arXiv：cs.LG · arxiv.org