arXiv：cs.LG· Yunied Puig, Amit Kumar Jaiswal·· 4 小时前AI 评分36
OrBIT：结构引导的嵌入向量压缩框架
OrBIT: Structure-Guided Embedding Compression
AI 导读
OrBIT 是一个结构引导的嵌入向量压缩框架，从轨道动力学中学习可复用的局部几何结构来约束共享码字，并让全局重建残差决定编码预算分配。在四个 LLM 嵌入向量表上，OrBIT 相对 16-bit 存储对 GPT-2 实现 37.9 倍压缩，每个 7B 表压缩超 23 倍，率失真性能可与现有量化和低秩基线竞争。
正文
Abstract:Embedding tables are among the largest components of modern language models. Most compression methods fix a coding geometry such as coordinate blocks, low-rank subspaces, or unrestricted codebooks, and optimize within it. We instead ask whether the coding geometry can itself be discovered. We introduce \emph{OrBIT}, a structure-guided embedding compression framework that learns reusable local geometry from orbit dynamics and uses it to constrain a small set of shared codewords. The global reconstruction residual then decides where the fixed coding budget is spent, while redundant overlapping charts let local errors compensate one another after gluing. Our theory shows how tight-chart geometry controls distortion, how the global residual directs sequential allocation, and how data-geometry-guided refinement improves the codec. The resulting orbit machinery is compiled away, leaving a compact decoder in which the learned structure governs what is stored, where capacity is allocated, and how local information is assembled globally. Across four LLM embedding tables, OrBIT achieves $37.9\times$ compression on GPT-2 and over $23\times$ on each 7B table relative to 16-bit storage, while delivering competitive rate-distortion performance against established quantization and low-rank baselines.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.10385 [cs.LG] |
| (or arXiv:2610.10385v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10385 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yunied Puig De Dios [view email]
[v1]
Wed, 7 Oct 2026 16:44:47 UTC (124 KB)
来源：arXiv：cs.LG · arxiv.org