arXiv：cs.LG（机器学习，全量分类）· Weili Xu, Jisen Li, Yuqing Jian, Chenxi Li, Zhizhou Sha, Yifan Yu, Qingyang Wu, Chenfeng Xu, Zhongzhu Zhou, Tianyi Zhang, Ben Athiwaratkun·· 15 小时前AI 评分44
QATFactory：面向 LLM 量化感知训练与蒸馏的部署对齐框架
QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs
AI 导读
开源框架 QATFactory 支持部署对齐的量化感知蒸馏（QAD）与强化学习（QARL），可在不支持 FP4 Tensor Core 的 H100 上以 BF16 矩阵乘法模拟 NVFP4 训练。
正文
Authors:Weili Xu, Jisen Li, Yuqing Jian, Chenxi Li, Zhizhou Sha, Yifan Yu, Qingyang Wu, Chenfeng Xu, Zhongzhu Zhou, Tianyi Zhang, Ben Athiwaratkun
Abstract:Large language model (LLM) inference is increasingly moving toward lower precision to realize the throughput of hardware accelerators, but aggressive post-training quantization (PTQ) can degrade model quality. We present QATFactory, an open-source framework for deployment-aligned quantization-aware distillation (QAD) and reinforcement learning (QARL). QATFactory simulates deployment-time quantization while performing matrix multiplications in BF16, allowing models to adapt to quantization noise without requiring training hardware that natively supports the target format; for example, it supports NVFP4 training on H100 GPUs, which lack FP4 Tensor Cores. The framework supports NVFP4, MXFP4, and $\text{llama}.\text{cpp}$'s Q4_K format; dense and mixture-of-experts models; and both full-parameter and LoRA-based training. It exports checkpoints directly to vLLM and $\text{llama}.\text{cpp}$ without an additional lossy conversion step or added inference overhead. With QATFactory, we conduct extensive experiments on models ranging from 8B to 230B parameters and evaluate exported checkpoints in production inference engines. Across models and formats, QAD consistently improves deployed-model quality over strong PTQ baselines. On Qwen3.5-9B, QAD achieves average benchmark accuracies of 68.9% under NVFP4 and 66.0% under MXFP4, outperforming the best PTQ results of 65.4% and 56.4%, respectively. Through our experiments, we found that although both FP4 formats quantize weights and activations at deployment, the best training strategy is format-dependent: NVFP4 generally performs better when only weights are quantized during training, whereas MXFP4 benefits from quantizing both weights and activations. At a fixed training token budget, training on fewer 32K sequences improves average accuracy by 1.9 points over training on more 4K sequences.
| Comments: | Preprint, code at this https URL |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.39223 [cs.LG] |
| (or arXiv:2609.39223v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.39223 arXiv-issued DOI via DataCite |
Submission history
From: Weili Xu [view email]
[v1]
Wed, 30 Sep 2026 08:00:35 UTC (937 KB)
[v2]
Thu, 1 Oct 2026 02:58:50 UTC (937 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org