arXiv：cs.AI· Rui Fang, Hsi-Wen Chen, Chia-Ming Cheng, Ming-Syan Chen·· 22 小时前
LoopQ：面向循环语言模型的量化框架
LoopQ: Quantization for Looped Language Models
AI 导读
针对循环语言模型（LoopLMs）对训练后量化（PTQ）高度敏感的问题，研究者提出 LoopQ 框架，通过循环感知激活缩放、跨循环状态对齐和轨迹感知校准，在 W4A4 量化下于四个骨干网络上实现五任务平均准确率 67.8% 的相对提升，八个骨干-数据集组合的平均困惑度相对降低保守估计达 75.2%。
正文
Abstract:Looped language models (LoopLMs) increase effective computational depth by repeatedly applying shared Transformer blocks without increasing the parameter count. However, this reuse makes them particularly sensitive to post-training quantization (PTQ). We systematically study PTQ in LoopLMs and identify three sources of degradation: distribution shift across roles, state reuse across loop transitions, and recursive error accumulation. We propose LoopQ, a loop-aware PTQ framework that preserves a shared quantized backbone while introducing lightweight adaptations. LoopQ combines loop-aware activation scaling and selective transformations to accommodate changing hidden-state distributions, cross-loop state alignment to stabilize transitions, and trajectory-aware calibration to limit accumulated error. Under W4A4 quantization, LoopQ achieves a 67.8% average relative improvement in five-task mean accuracy across four backbones and a conservative estimate of 75.2% for the average relative perplexity reduction across eight backbone-dataset pairs, compared with the strongest static PTQ baseline for each comparison.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.16343 [cs.LG] |
| (or arXiv:2605.16343v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.16343 arXiv-issued DOI via DataCite |
Submission history
From: Rui Fang [view email]
[v1]
Fri, 8 May 2026 01:45:28 UTC (485 KB)
[v2]
Mon, 12 Oct 2026 16:51:06 UTC (242 KB)
来源：arXiv：cs.AI · arxiv.org