arXiv：cs.LG· Yiping Ji, James Martens, Simon Lucey·· 4 小时前AI 评分38
DDT-RFE：跳过扩散模型中的残差连接？
Should We Skip Diffusion?
AI 导读
研究提出 DDT-RFE，移除 Decoupled Diffusion Transformer 编码器块中 Self-Attention 和 MLP 周围的残差连接，并将输入 patch embedding 与中间及最终编码器特征融合，使每个编码块学到更抽象表示的同时让解码器获取多层深度信息。
正文
Abstract:Diffusion models learn semantic representations while generating images. In the Decoupled Diffusion Transformer (DDT), a condition encoder provides features that guide a velocity decoder in denoising. To enable effective denoising at all noise levels, these features must capture both high-level abstract structures and low-level details. However, skip/residual connections in the encoder allow shallow features to bypass successive transformations, which may limit progressive abstraction, or at least make it difficult to disentangle different levels of abstraction. We propose DDT-RFE, which removes the residual connections around the Self-Attention and MLP operations in each encoder block while maintaining stable training. To retain the information that abstraction discards but that the decoder still needs, we fuse the input patch embedding with intermediate and final encoder features to form the encoder output. The decoder thus has access to information from multiple encoder depths, while each encoder block is able to learn more abstract representations. DDT-RFE achieves overall improvements over DDT across visual understanding tasks, including image classification, semantic segmentation, object discovery, and semantic correspondence, while using fewer encoder blocks. It also achieves a lower FID for image generation on ImageNet.
| Subjects: | Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2610.07002 [cs.LG] |
| (or arXiv:2610.07002v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07002 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yiping Ji [view email]
[v1]
Sun, 4 Oct 2026 09:42:27 UTC (7,339 KB)
来源：arXiv：cs.LG · arxiv.org