HuggingFace Daily Papers·· 2 天前
SpecMRL：在参数化频谱中重新设计 Matryoshka 表示学习
SpecMRL: Redesign Matryoshka Representation Learning within Parametric Spectrum
AI 导读
SpecMRL 通过引入按特征重要性组织的参数化频谱基，在学习的频谱空间中做前缀读取，从而改进 MRL 的嵌入压缩效果。MTEB 实验显示，SpecMRL 在所有评估的降维设置下都优于 MRL，压缩越激进增益越大；仅用一半嵌入维度即可超过全维度的 MRL。
正文 · 原文
Abstract:Matryoshka Representation Learning (MRL) is designed to support flexible representation sizes by jointly optimizing nested prefixes. While effective, this design lacks an explicit constraint to ensure that the leading portion provides a suitable compact encoding. To address this, we propose SpecMRL, which reorganizes the feature space to enable more effective prefix readout. Specifically, we extend MRL with a parameter-derived spectral basis organized with feature importance. SpecMRL otherwise follows standard MRL, with prefix readout performed in the learned spectral space. Experiments on MTEB show that SpecMRL outperforms MRL at all evaluated reduced dimensions, with larger gains under more aggressive compression. These results suggest that the learned spectral basis better prioritizes informative directions, enabling more effective representation compression. Notably, SpecMRL extends the performance frontier of MRL, surpassing full-dimensional MRL with only half the embedding dimensions.
| Comments: | preprint |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.26824 [cs.CL] |
| (or arXiv:2610.26824v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.26824 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Songhao Wu [view email]
[v1]
Mon, 19 Oct 2026 17:54:20 UTC (414 KB)
来源：HuggingFace Daily Papers · arxiv.org