扩散模型的结构性记忆：超越样本复制

View PDF HTML (experimental)

Abstract:Diffusion models generalize well in practice. Paradoxically, an optimal diffusion model fully memorizes the training data and therefore fails to generalize, raising the question of what induces generalization in a real diffusion model. We show that diffusion models progressively overfit the denoising training objective, creating a generalization gap between validation and training performance at intermediate noise levels. In a fully analytic 2D toy model with a controlled denoising error, we trace this gap to the interaction between model error and the density of the data distribution's support. The optimal denoising flow field localizes sharply around individual training points, whereas model error suppresses exact recall of training points, yielding a smooth, generalizing flow field. Finally, we examine how training-time overfitting manifests along inference trajectories. We find that predictions made from intermediate trajectory states occupy a distinct feature-space regime from predictions made from noised training and validation images. As training progresses and model size increases, these predictions develop greater relative affinity to the training data, despite the absolute similarity to the validation data not decreasing. Together, these findings show that the denoising objective and inference trajectories express structural memorization differently.

Submission history

From: Tim Kaiser [view email]
[v1] Thu, 12 Mar 2026 21:02:17 UTC (12,336 KB)
[v2] Wed, 20 May 2026 10:08:19 UTC (12,740 KB)
[v3] Thu, 8 Oct 2026 09:39:08 UTC (4,771 KB)