arXiv：cs.CL· Saibo Geng, Andrea Gentilini, Xinxian Ma, Nathan Ranchin, Yunzhen Yao, Robert West·· 13 小时前
zip2zip++：在推理时实现近无损的 token 压缩
zip2zip++: Towards Inference-Time Token Compression with Minimal Quality Loss
AI 导读
zip2zip++ 通过改进压缩层与 LZW 编解码器，在不修改预训练语言模型的前提下，将 zip2zip 与未压缩基线之间的质量差距缩小约一半，压缩比相近。研究识别出共享嵌入冲突、RoPE 位置处理、数字合并损伤准确率及动态词表分割歧义四个问题，并分别用解耦输入/输出动态嵌入层、size-aware RoPE 和改进 LZW 编解码器解决。
正文
Abstract:Dynamic vocabularies built on-the-fly via Lempel-Ziv-Welch (LZW) compression let a language model represent the same text in fewer tokens, but prior work shows that this efficiency comes at a cost to reasoning and perplexity. We show that this quality gap is not a fundamental price of compression but can be substantially closed through targeted improvements to the compression layer and LZW codec, without modifying the underlying pretrained language model. Our analysis reveals four key issues: (1) LZW's compression algorithm creates an implicit conflict between the input embedding and the output embedding, a tension that a single shared embedding cannot resolve; (2) naively applying RoPE to LZW-compressed sequences ignores that merged tokens can span multiple tokens; (3) merging certain tokens (notably digits) disproportionately hurts downstream accuracy; and (4) dynamic vocabularies introduce an inherent segmentation ambiguity that confounds standard evaluation. We address these through decoupled input/output dynamic embedding layer, size-aware RoPE positioning, and a modified LZW codec supporting selective merges and multi-view segmentations. Across four models, our recipe improves perplexity on every corpus we evaluated as well as mathematical reasoning accuracy, closing roughly half the quality gap between zip2zip and the uncompressed baseline at a similar compression ratio on general text. This represents a significant step toward lossless adaptive-vocabulary compression for language models.
| Comments: | 27 pages, 12 figures, 13 tables. Code: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.13605 [cs.CL] |
| (or arXiv:2610.13605v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13605 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Saibo Geng [view email]
[v1]
Fri, 9 Oct 2026 13:52:54 UTC (721 KB)
来源：arXiv：cs.CL · arxiv.org