MPZCH：基于多探针零碰撞哈希的大规模推荐系统嵌入碰撞缓解方案
Multi-Probe Zero Collision Hash (MPZCH): Mitigating Embedding Collisions and Enhancing Model Freshness in Large-Scale Recommenders
研究者提出 Multi-Probe Zero Collision Hash（MPZCH），一种基于线性探测的索引机制，可在合理表规模下完全消除嵌入碰撞，并保持生产级效率。
Authors:Ziliang Zhao, Bi Xue, Emma Lin, Tianqi Lu, Mengjiao Zhou, Kaustubh Vartak, Shakhzod Ali-Zade, Tao Li, Bin Kuang, Rui Jian, Bin Wen, Dennis van der Staay, Yixin Bao, Xiujin Li, Chao Deng, Henry Wei, Songbin Liu, Qifan Wang, Kai Ren
Abstract:Embedding tables are critical components of large-scale recommendation systems, facilitating the efficient mapping of high-cardinality categorical features into dense vector representations. However, as the volume of unique IDs expands, traditional hash-based indexing methods suffer from collisions that degrade model performance and personalization quality. We present Multi-Probe Zero Collision Hash (MPZCH), a novel indexing mechanism based on linear probing that effectively mitigates embedding collisions. With reasonable table sizing, it often eliminates these collisions entirely while maintaining production-scale efficiency. MPZCH utilizes auxiliary tensors and high-performance CUDA kernels to implement configurable probing and active eviction policies. By retiring obsolete IDs and resetting reassigned slots, MPZCH prevents the stale embedding inheritance typical of hash-based methods, ensuring new features learn effectively from scratch. Despite its collision-mitigation overhead, the system maintains training QPS and inference latency comparable to existing methods. Rigorous online experiments demonstrate that MPZCH achieves zero collisions for user embeddings and significantly improves item embedding freshness and quality. The solution has been released within the open-source TorchRec library for the broader community.
| Comments: | 10 pages, 6 figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2602.17050 [cs.LG] |
| (or arXiv:2602.17050v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.17050 arXiv-issued DOI via DataCite |
|
| Journal reference: | Proceedings of the 20th ACM Conference on Recommender Systems (RecSys '26), 2026, pp. 1333-1342 |
| Related DOI: | https://doi.org/10.1145/3773078.3831866
DOI(s) linking to related resources |
Submission history
From: Ziliang Zhao [view email]
[v1]
Thu, 19 Feb 2026 03:42:57 UTC (833 KB)
[v2]
Mon, 23 Feb 2026 21:14:23 UTC (833 KB)
[v3]
Thu, 14 May 2026 18:35:53 UTC (833 KB)
[v4]
Tue, 6 Oct 2026 16:54:07 UTC (860 KB)
来源：arXiv：cs.LG · arxiv.org