arXiv：cs.CL· David Porte\v{s}, Ale\v{s} Hor\'ak·· 3 小时前
Prosody-to-Text：从低通滤波语音中预测文本
Prosody-to-Text: Predicting text from low-pass filtered speech
AI 导读
研究者提出"韵律到文本"任务，用仅保留 12 个最低 Mel 频段（约 450Hz 截止的低通滤波）微调 Whisper，从韵律模式中恢复原始句子，词错误率（WER）为 36%，10% 的语句被完美还原，40% 的语句 WER 不超过 25%。在给定正确前缀时，下一 token 预测正确率达 79%。结果提示低频语音特征与词汇内容的关联比此前认为的更强，或可启发用韵律引导现代 LLM 的文本生成。
正文
Abstract:While predicting prosody from text is an established task in the field, the opposite direction, predicting text that fits a given prosodic pattern, remains largely overlooked. We find this unfortunate, because this opposite direction could lead to some very interesting use cases. Therefore, in this paper, we make the first steps in the prosody-to-text direction by inves- tigating how much of the original sentence can be recovered from its prosodic pattern. To this end, we fine-tune the Whis- per model using only the 12 lowest Mel bins (low-pass filter with approximately 450Hz cutoff), and obtain surprisingly accurate results (WER 36%), with 10% of utterances be- ing recovered perfectly, and 40% of utterances having Word Error Rate at or below 25%. We also find that, given the correct prefix, the next token was predicted correctly in 79% of cases. Our results suggest that the relationship between low-frequency speech features and lexical content is much stronger than previously thought, and we believe that direct- ing more attention to this topic might open the door to new applications, such as using prosody to guide text generation of modern LLMs
| Comments: | This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.11544 [cs.CL] |
| (or arXiv:2610.11544v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11544 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: David Porteš [view email]
[v1]
Thu, 8 Oct 2026 09:12:02 UTC (496 KB)
来源：arXiv：cs.CL · arxiv.org