arXiv：cs.LG· Alessandro Beatini, Marco Maronese, Emanuele Rodol\`a·· 4 小时前AI 评分33
AWT：面向张量网络 LLM 压缩的激活感知权重张量化预条件方法
Activation-Aware Weight Tensorization: A Calibration-Time Preconditioner for Tensor-Network LLM Compression
AI 导读
研究提出 Activation-aware Weight Tensorization（AWT），一种免训练的校准封装，在 TT/TTN 分解前用激活导出的对角缩放对权重矩阵做预条件，部署时仅需输入端逐元素重缩放。
正文
Abstract:Post-training tensor-network compression replaces Transformer linear layers with Tensor Train (TT) or Tree Tensor Network (TTN) operators, but standard decompositions minimize weight-space Frobenius error rather than functional error under the layer's activation distribution. We propose Activation-aware Weight Tensorization (AWT), a training-free calibration wrapper that preconditions each weight matrix with a diagonal activation-derived scale before an unchanged TT/TTN solver and deploys the result with only an input-side elementwise rescaling. Across Llama 3.1 8B, Ministral 8B, and Qwen2.5 7B, AWT consistently improves vanilla TT/TTN tensorization at 2-6 times compression: under single-operator replacement, AWT closes 12-35% of the WikiText perplexity gap to the dense baseline across the three model families and 2-6 times compression settings; while under multi-operator Llama suffix replacement it closes 27-60% across attention-group and all-seven-matrix settings. The gains also transfer to downstream HellaSwag and ARC-Challenge evaluations. We further show that diagonal preconditioning is a robustness-modularity tradeoff rather than a diagonal-covariance assumption: a dense full-covariance oracle wins its own weighted objective in 80/81 cases, yet diagonal AWT gives better held-out functional fidelity in 53/81 cases. Together, these results position AWT as a principled, modular preconditioner for improving functional fidelity in fixed TT/TTN compression pipelines without modifying the decomposition solver.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.10085 [cs.LG] |
| (or arXiv:2610.10085v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10085 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Alessandro Beatini [view email]
[v1]
Wed, 7 Oct 2026 13:48:04 UTC (27 KB)
来源：arXiv：cs.LG · arxiv.org