arXiv：cs.LG（机器学习，全量分类）· Shixuan Liu, Tongli Zhou, Junwei Deng, Pingbang Hu, Jiaqi W. Ma·· 5 小时前AI 评分40
dattri-LLM：面向 LLM 规模训练数据归因的统一高效库
dattri-LLM: A Unified and Efficient Library for Training Data Attribution at LLM Scale
AI 导读
dattri-LLM 是一个面向 LLM 规模的训练数据归因（TDA）库，通过紧凑梯度表示和基于成本模型的动态梯度操作路由提升效率。它无需修改训练循环即可从调用 backward() 的现有流程中采集逐样本梯度，兼容 DDP、FSDP 及 HuggingFace Transformers、TRL、OLMo 构建的流水线。
正文
Abstract:Training data attribution (TDA) estimates the contribution of individual training examples to model outputs. Most scalable TDA methods rely on per-example gradients, whose computation and use at LLM scale pose challenges in efficiency, compatibility, and extensibility. We introduce dattri-LLM, a TDA library that makes gradient-based attribution more practical at scale. For efficiency, dattri-LLM uses compact gradient representations and dynamically routes gradient operations based on a cost model. For compatibility, its capture mechanism collects per-example gradients from existing training loops that call backward(), without requiring changes to the loop or its configuration. This includes distributed training with DDP and FSDP and pipelines built with HuggingFace Transformers, TRL, and OLMo. For extensibility, dattri-LLM exposes reusable gradient operations and training-time callbacks for implementing attribution methods and applications. These interfaces support a variety of attribution methods, including gradient similarity, curvature-based influence, and trajectory-based methods, as well as applications that act on gradients during training, such as online data selection. On the same hardware and workload, dattri-LLM achieves 3.2x the throughput of the fastest competing library on average, scales multiple attribution methods to 110B-parameter models across four H200 GPUs, and offers superior attribution fidelity-cost trade-offs across a range of models with different model families and scales. The source code of dattri-LLM is available at this https URL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.38767 [cs.LG] |
| (or arXiv:2609.38767v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.38767 arXiv-issued DOI via DataCite |
Submission history
From: Shixuan Liu [view email]
[v1]
Wed, 30 Sep 2026 01:49:42 UTC (330 KB)
[v2]
Thu, 1 Oct 2026 16:36:54 UTC (330 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org