ARIA：面向粤语歌词创作、从演唱音频建模旋律与声调关系的框架

View PDF HTML (experimental)

Abstract:Cantonese lyric writing requires close alignment between lexical tones and melodic pitch. Existing melody-guided lyric generation methods typically rely on symbolic melody to generate lyrics. However, in real songwriting scenarios, melodies are often expressed as raw singing audio or hummed recordings, where pitch is implicit, noisy, and unstructured, making these methods difficult to apply directly. To address this limitation, we propose ARIA, a two-stage audio-driven melody-tone relation modeling framework for Cantonese lyric authoring that generates Cantonese lyrics from singing recordings with provided character-level timestamps. Specifically, we first design a Tri-Stream Relation-Aware Tone Estimator (TRATE) to predict 0243 sequences from timestamped singing audio by modeling multi-stream acoustic cues and relational tonal structure. We then propose a Decoupled Retrieval-Augmented Tone-Conditioned Lyric Generator (DRA-TCLG) to generate fluent lyrics conditioned on predicted tonal plans with retrieval-enhanced lexical guidance. Moreover, we construct a large-scale aligned audio-Jyutping-0243 dataset from real Cantonese singing recordings to support this new task. Experimental results demonstrate that ARIA achieves strong performance in both 0243 prediction and tone-consistent lyric generation, validating the effectiveness of the proposed framework.

Comments:	Accepted for publication in Findings of EMNLP 2026. 24 pages, including references and appendices. Author-prepared version
Subjects:	Computation and Language (cs.CL); Sound (cs.SD)
Cite as:	arXiv:2610.07902 [cs.CL]
	(or arXiv:2610.07902v1 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2610.07902 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Shengyu Li [view email]
[v1] Tue, 6 Oct 2026 07:48:41 UTC (544 KB)