arXiv：cs.AI· Jincheng Zhang, Gy\"orgy Fazekas, Charalampos Saitis·· 10 小时前
扩散模型自表征对齐：U-SRA 用于视频背景音乐生成
Self-Representation Alignment in Diffusion Models for Video Background Music Generation
AI 导读
研究者提出扩散 U-Net 自表征对齐（U-SRA）架构，在训练中将高噪声条件下较弱的早期层表征与低噪声条件下较好的后期层表征显式对齐，并采用对齐损失引导模型学习更好的内部表征。该方法用于视频背景音乐生成任务，通过 cross-attention 机制使音乐与视频片段同步。在基准视频-音乐数据集上，U-SRA 在视频-音乐对齐和整体音乐质量等关键指标上超越了现有最优方法。
正文
Abstract:Diffusion models have been widely used for various generative tasks. However, recent studies have demonstrated that the internal representations learned in the diffusion models' denoising process are insufficiently meaningful, which can hinder the generation performance. To address this, we propose a novel diffusion U-Net architecture enhanced by explicitly aligning the weaker representation from the earlier layer conditioned on higher noise with the better representation from the later layer conditioned on lower noise during training, which we use for the challenging multimodal task of video background music generation. Our diffusion U-Net Self-Representation Alignment (U-SRA) employs an alignment loss to effectively guide our diffusion model to learn good internal representations, thereby enabling it to produce higher-quality symbolic music conditioned on video features. We employ a cross-attention mechanism to synchronize music with video segments. Experiments on a benchmark video-music dataset demonstrate that the proposed U-SRA approach for video background music generation surpasses the state-of-the-art across key evaluation metrics, including video-music alignment and overall music quality.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15575 [cs.AI] |
| (or arXiv:2610.15575v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15575 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jincheng Zhang [view email]
[v1]
Sun, 11 Oct 2026 12:40:40 UTC (1,080 KB)
来源：arXiv：cs.AI · arxiv.org