AlignAudio：面向噪声鲁棒语音条件生成的统一双重对齐框架

View PDF HTML (experimental)

Abstract:Speech-conditioned generation maps speech signals to target audio outputs. Prior end-to-end speech-conditioned generative models achieve high fidelity in clean acoustic conditions, but their performance degrades substantially under environmental noise. While prior noise-robust approaches have achieved significant success in feature-level purification, maintaining consistent generation under clean and noisy conditioning remains a key challenge. To address this, we propose AlignAudio, a noise-robust generative framework based on dual alignment: (i) aligning the representations of clean and noisy speech to preserve semantic cues, and (ii) enforcing generation-level consistency within a flow matching paradigm to improve conditioning fidelity. Experiments on speech-to-audio (STA) and voice conversion (VC) tasks demonstrate that AlignAudio maintains performance comparable to baselines on clean speech while significantly outperforming them across various noisy conditions. Furthermore, the proposed dual-alignment strategy is applicable to both single- and multi-stream speech conditioning and can be integrated into different flow matching-based generation frameworks and post-training settings. Audio demonstrations and qualitative comparisons: \href{this https URL}{\textcolor{cyan}{\textit{this https URL}}}.

Submission history

From: Yixuan Li [view email]
[v1] Sat, 10 Oct 2026 08:15:57 UTC (5,375 KB)