arXiv：cs.CL· Yaroslav Getman, Tam\'as Gr\'osz, Tommi Lehtonen, Mikko Kurimo·· 13 小时前
从 2.1M 小时广播音频到芬兰语语音 SSL：置信度过滤与大规模架构分析
From 2.1M Hours of Raw Broadcast Audio to Finnish Speech SSL: Confidence Filtering and Architectural Analysis at Scale
AI 导读
一项芬兰语语音 SSL 研究以 2.1M 小时无约束广播音频为起点，提出用 ASR 置信度作为预训练数据质量的代理指标，在训练集更小的情况下比基于 LID 的过滤取得 3-7% 的相对 WER 改善。研究还在匹配参数预算下对比了 Transformer、Macaron-Net、Conformer 和 E-Branchformer，模型在 Base 规模和域外表现上优于此前的芬兰语 SSL 基线。
正文
Abstract:Broadcast archives offer scale and diversity for self-supervised learning (SSL) but contain substantial noisy, non-speech, and multilingual content. Prior work typically relies on voice activity detection or language identification (LID) to extract speech, ensuring presence rather than quality. Whether and how to filter unlabeled speech by quality for SSL, particularly from noisy broadcasts, remains underexplored. We present a large-scale Finnish SSL study starting from 2.1M hours of unconstrained broadcast audio. We propose automatic speech recognition (ASR) confidence as a proxy for pretraining data quality and show 3-7% relative WER improvements over LID-based filtering despite a smaller training set. We further address the lack of controlled architectural comparisons in speech SSL by evaluating Transformer, Macaron-Net, Conformer, and E-Branchformer at matched parameter budgets. Our models improve upon prior Finnish SSL baselines, particularly at Base scale and out-of-domain.
| Comments: | Accepted to SLT 2026 |
| Subjects: | Computation and Language (cs.CL); Sound (cs.SD); Audio and Speech Processing (eess.AS) |
| Cite as: | arXiv:2610.13102 [cs.CL] |
| (or arXiv:2610.13102v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13102 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yaroslav Getman [view email]
[v1]
Fri, 9 Oct 2026 06:57:05 UTC (132 KB)
来源：arXiv：cs.CL · arxiv.org