arXiv：cs.LG· Maximilian B\"other, Josh Wills, Ties Robroek, Sonnet Xu, Paul Burstein, Daniel Zayas, Cody Blakeney, Siddharth Joshi, Haoli Yin, Rishabh Adiga, Haakon Mongstad, Luke Merrick, Pratyush Maini, Ari Morcos, Matthew Leavitt, Ana Klimovic, Bogdan Gaza·· 3 小时前AI 评分39
Zephon：面向在线有状态基础模型数据加载管道的弹性确定性方案
Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines
AI 导读
研究者提出 Zephon，一个面向基础模型的数据加载器，可在 GPU 拓扑变化、频繁 checkpoint-resume 和不同数据处理后端下，仍保证全局训练数据批次的确定性序列。
正文
Authors:Maximilian Böther, Josh Wills, Ties Robroek, Sonnet Xu, Paul Burstein, Daniel Zayas, Cody Blakeney, Siddharth Joshi, Haoli Yin, Rishabh Adiga, Haakon Mongstad, Luke Merrick, Pratyush Maini, Ari Morcos, Matthew Leavitt, Ana Klimovic, Bogdan Gaza
Abstract:Deterministic data loading is important for foundation model development: model researchers need confidence that differences they observe across costly ablations are caused by the parameter they changed rather than non-determinism in the training data sequence. The data loader must provide elastic determinism, i.e., a deterministic sequence of global training data batches despite changes to the GPU topology across runs (e.g., due to GPU scarcity), frequent checkpoint-resume cycles, and different data processing execution backends. Achieving this is difficult because modern foundation model data pipelines tokenize, pack, and mix samples online, introducing stateful n-to-m transformations that break sample indexing. Existing data loaders largely assume indexable 1-to-1 pipelines, and the common workaround of offline materialization is expensive and, for some modalities such as video, infeasible.
We present Zephon, a data loader for foundation models that supports online, stateful pipelines while providing elastic determinism and efficient resumption from checkpoints. It partitions the global stream into topology-independent lanes, serializes ordering decisions while parallelizing stateless work on interchangeable backends, and checkpoints only bounded in-flight state so recovery cost does not grow with training progress. We evaluate Zephon on text and vision-language workloads and show that it achieves competitive throughput while providing a combination of guarantees that no existing loader offers for online, stateful pipelines.
| Comments: | preprint; currently under revision at VLDB'27 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Databases (cs.DB) |
| Cite as: | arXiv:2610.03087 [cs.LG] |
| (or arXiv:2610.03087v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.03087 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Maximilian Böther [view email]
[v1]
Fri, 2 Oct 2026 10:06:32 UTC (531 KB)
来源：arXiv：cs.LG · arxiv.org