用条件流匹配（CFM）实现视觉引导的音频高亮

View PDF HTML (experimental)

Abstract:Visually-guided acoustic highlighting seeks to rebalance audio in alignment with the accompanying video, creating a coherent audio-visual experience. While visual saliency and enhancement have been widely studied, acoustic highlighting remains underexplored, often leading to misalignment between visual and auditory focus. Existing approaches use discriminative models, which struggle with the inherent ambiguity in audio remixing, where no natural one-to-one mapping exists between poorly-balanced and well-balanced audio mixes. To address this limitation, we reframe this task as a generative problem and introduce a Conditional Flow Matching (CFM) framework. A key challenge in iterative flow-based generation is that early prediction errors -- in selecting the correct source to enhance -- compound over steps and push trajectories off-manifold. To address this, we introduce a rollout loss that penalizes drift at the final step, encouraging self-correcting trajectories and stabilizing long-range flow integration. We further propose a conditioning module that fuses audio and visual cues before vector field regression, enabling explicit cross-modal source selection. Extensive quantitative and qualitative evaluations show that our method consistently surpasses the previous state-of-the-art discriminative approach, establishing that visually-guided audio remixing is best addressed through generative modeling.

Subjects:	Audio and Speech Processing (eess.AS); Machine Learning (cs.LG)
Cite as:	arXiv:2602.03762 [eess.AS]
	(or arXiv:2602.03762v5 [eess.AS] for this version)
	https://doi.org/10.48550/arXiv.2602.03762 arXiv-issued DOI via DataCite

Submission history

From: Hugo Malard [view email]
[v1] Tue, 3 Feb 2026 17:24:47 UTC (15,561 KB)
[v2] Wed, 4 Feb 2026 08:53:22 UTC (15,561 KB)
[v3] Mon, 22 Jun 2026 14:57:28 UTC (29,954 KB)
[v4] Wed, 24 Jun 2026 20:18:09 UTC (24,878 KB)
[v5] Wed, 7 Oct 2026 20:31:57 UTC (24,381 KB)