arXiv：cs.LG· Seobin Song, Geonho Lee, Janghwan Lee, Jungwook Choi·· 4 小时前AI 评分39
Align, Then Correct：面向极低比特量化大语言模型的无训练两阶段低秩补偿方法
Align, Then Correct: Training-Free Two-Stage Low-Rank Compensation for Extremely Quantized Large Language Models
AI 导读
研究者提出无训练两阶段闭式框架，改进低秩量化误差补偿（LQEC）中对称校准和高秩补偿目标两大简化问题。在 2 bits、QuIP# 量化下，该方法将 Qwen3-8B 的 WikiText-2 困惑度从 12.43 降至 10.26，Qwen3-4B 从 21.11 降至 13.22；在 C4 语料上对 FP16 差距的恢复率分别为 51% 和 84%，优于最强基线的 31% 和 63%。
正文
Abstract:Low-rank quantization error compensation (LQEC) recovers the accuracy lost under aggressive weight quantization by attaching a closed-form rank-$r$ adapter beside each frozen quantized weight, without any training. We show that existing compensators are limited by two shared simplifications. They calibrate symmetrically, evaluating the full-precision and compensated weights on the same activation, which yields a compensation target that is inherently high-rank -- so a fixed rank budget captures only a small fraction of it. And they minimize only the second-order term of the loss, although the compensated model is not stationary: a first-order descent direction larger than the applied compensation itself remains in every layer, and no reconstruction objective can absorb it. We propose a two-stage closed-form framework that removes both simplifications. Stage 1 aligns each layer's output with the full-precision model under a Fisher-weighted asymmetric objective, concentrating the rank budget on a rank-compressible target. Stage 2 re-measures statistics on the compensated model and applies a rank-constrained natural-gradient step that absorbs the remaining first-order signal. Every adapter is the result of a single truncated SVD; backward passes serve only to collect statistics. At 2 bits under QuIP#, our method reduces WikiText-2 perplexity from 12.43 to 10.26 on Qwen3-8B and from 21.11 to 13.22 on Qwen3-4B. On the held-out C4 corpus, it recovers 51% and 84% of the gap to FP16, versus 31% and 63% for the strongest baseline, with consistent gains in the seven-task zero-shot average, at higher bit-widths, and under a distinct quantizer.
| Comments: | 17 pages, 5 figures |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.08164 [cs.LG] |
| (or arXiv:2610.08164v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.08164 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jungwook Choi [view email]
[v1]
Tue, 6 Oct 2026 11:16:37 UTC (1,365 KB)
来源：arXiv：cs.LG · arxiv.org