arXiv：cs.LG· Alvard Barseghyan, Ani Vanyan, Hakob Tamazyan, Hrant Khachatrian·· 3 小时前
图像自蒸馏的数据密度缩放定律：重复样本有时胜过更多唯一样本
Data Density Scaling Laws for Image Self-Distillation
AI 导读
针对图像自蒸馏模型，在 Maxar、SATLAS、Waymo、nuPlan、Ego4D、ImageNet 和 YFCC15M 上的系统实验显示，重复样本对下游性能的影响因数据集而异，在某些设置下反复训练较小的随机子集反而优于训练更大的唯一样本集。
正文
Abstract:Current scaling law formulations suggest that increasing the number of unique samples in a dataset improves model performance, while repeated exposure to the same samples provides diminishing value. In this work, we examine this assumption for image self-distillation models. Through systematic experiments across diverse datasets, namely Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M, we demonstrate that the effects of sample repetition on downstream performance are non-trivial and dataset-dependent. We identify settings where repeatedly training on a smaller random subset can outperform training on a larger set of unique samples. This finding highlights a limitation of existing scaling laws. To address this gap, we propose a modified version of a Chinchilla-style scaling law that explicitly accounts for two factors: the number of unique samples and repetition per sample. This formulation provides a framework for understanding data-compute trade-offs and improves prediction in some regimes where repetition plays a central role, although its benefits are not consistent across datasets. Finally, we examine how subset selection and augmentation affect downstream performance.
| Comments: | 25 pages, 11 figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.19013 [cs.LG] |
| (or arXiv:2610.19013v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19013 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hrant Khachatrian [view email]
[v1]
Tue, 13 Oct 2026 18:08:51 UTC (9,817 KB)
来源：arXiv：cs.LG · arxiv.org