arXiv：cs.CL· Vinko Sabol\v{c}ec, Bettina Messmer, Yassine Turki, Martin Jaggi·· 3 小时前
将英语质量分类器适配到多语言 LLM 预训练数据筛选
Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection
AI 导读
研究者提出一种多语言适配方法，将现有英语质量分类器扩展至 100 多种语言，用于 LLM 预训练数据筛选。该方法在 Transformer encoder-only 模型嵌入之上训练小型多层感知机，以机器翻译文本经英语分类器打分为标签。1B、3B 和 8B 规模实验显示，其下游 LLM benchmark 表现与现有多语言基于模型的筛选基线持平，且未损害区域与文化知识 benchmark。
正文
Abstract:Recent advances in large language model (LLM) pretraining highlight the role of high-quality training data in improving performance. While model-based filtering has proven effective in selecting high-quality subsets from web-scale corpora, especially for high-resource languages, low-resource languages face challenges due to limited availability of annotated data. This work explores extending quality filtering to over 100 languages by proposing a multilingual adaptation approach that converts an existing English quality classifier into a multilingual variant. Our approach proposes training a small multi-layer perceptron on top of Transformer encoder-only model embeddings, using multilingual text as input and scores obtained from English classifiers applied to machine-translated text as labels. Our 1B, 3B and 8B scale experiments show that our approach maintains the downstream LLM benchmark performance of existing multilingual model-based filtering baselines, without harming regional and cultural knowledge benchmarks. To further evaluate cross-lingual generalization, we compare classifier scores of high-quality synthetic data and web samples, and the correlation of classifier scores with LLM-based ones, revealing that the classifier can learn the scoring criteria of its original English variant, even for languages not included in its training data.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.11585 [cs.CL] |
| (or arXiv:2610.11585v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11585 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Vinko Sabolčec [view email]
[v1]
Thu, 8 Oct 2026 09:33:32 UTC (1,170 KB)
来源：arXiv：cs.CL · arxiv.org