arXiv：cs.AI· Luyang Zhang, Jialu Wang, Bebei Li·· 17 小时前
Pick the Mixture, Not the Samples：面向 LLM 的组合式数据选择框架与 Mixture-then-Select 算法
Pick the Mixture, Not the Samples: Combinatorial Data Selection for LLMs
AI 导读
针对 LLM 微调中数据池跨域、评测跨任务导致样本价值非可加的问题，研究者提出组合式数据选择（CDS）框架，将样本打分与子集选取分离，并据此设计两阶段算法 Mixture-then-Select：先决定各域采样数量，再按归因分数在域内排序。实验显示其比强基线高出 20% 至 54%，且在不同归因方法、模型和数据池规模下均稳定，该框架还可扩展到预算约束下的数据购买问题。
正文
Abstract:Data selection for LLM fine-tuning faces two facts about real training: data pools draw from many distinct domains, and evaluation spans diverse tasks. As a result, the same sample helps different tasks differently, and samples interact during training so that a subset's value is not the sum of its parts. Current data-selection methods leave these realities largely unaddressed. We formalize them as Combinatorial Data Selection (CDS), a framework that separates attribution (how samples are scored) from selection (which subset to pick) and models the non-additive interactions among samples. On a real instruction-tuning data pool, we find substantial cross-domain complementarity (joint utility exceeds the sum of singletons), alongside redundancy within domains. Inspired by this, we propose Mixture-then-Select, a two-stage algorithm that first decides how many samples to take from each domain, then ranks samples within each domain by attribution score. Theoretically, the gap to the optimal subset is linear in the selection size and independent of data pool size. Empirically, Mixture-then-Select outperforms strong baselines by 20 to 54%, is robust across attribution methods, and works across different models and data pool sizes. The framework also extends to the data purchasing problem under a budget constraint when sample prices vary. Our findings suggest that allocating selection across domains offers a promising path toward more efficient and effective data selection for modern LLMs.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14028 [cs.LG] |
| (or arXiv:2610.14028v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14028 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Luyang Zhang [view email]
[v1]
Wed, 30 Sep 2026 18:01:37 UTC (214 KB)
来源：arXiv：cs.AI · arxiv.org