arXiv：cs.LG· Yibo Huang, Yi Cui, Yukai Huang, Huaping Zhou, Jiancong Tong, Zhaoyu Wu, Gen Miao, Haoran Wei, Boyan Ding, Zhuo Jiang, Jianxi Ye, Yibo Zhu, Chuanxiong Guo, Yang Xu, Jie Wu·· 11 小时前
字节跳动如何用 ByteExpress 将 RDMA 部署到大规模推理与训练
Deploying RDMA for Large-Scale Inference and Training Workloads
AI 导读
字节跳动在基于参数服务器的推理和训练系统中部署了硬件 RDMA 之上的软件通信层 ByteExpress。生产推理中，模型参数拉取延迟均值与 P99 分别降低 35.7% 和 43.7%，CPU 总占用下降 10.69%；训练中参数推送 P99 延迟最高降低 74.0%，运行时未观察到 PFC 暂停。
正文
Authors:Yibo Huang, Yi Cui, Yukai Huang, Huaping Zhou, Jiancong Tong, Zhaoyu Wu, Gen Miao, Haoran Wei, Boyan Ding, Zhuo Jiang, Jianxi Ye, Yibo Zhu, Chuanxiong Guo, Yang Xu, Jie Wu
Abstract:RDMA can reduce communication overhead in distributed machine learning systems, but realizing these benefits in production requires more than replacing TCP. Variable-size messages, concurrent connections, bursty incasts, and service failures introduce challenges that hardware offload alone cannot address. This paper reports our experience deploying ByteExpress, a software communication layer over hardware RDMA, in ByteDance's parameter-server-based inference and training systems. We present three lessons from the deployment: efficient RDMA requires buffer management across application and transport layers; scalable communication requires coordinated management of connections, receive capacity, and outstanding data; and service availability requires recovery above the transport layer. ByteExpress combines an RDMA-aware allocator, zero-copy transfers, QP multiplexing, receive-credit flow control, a software send window, and RPC-assisted recovery. In production inference, ByteExpress reduces mean and P99 model parameter fetch latency by 35.7% and 43.7%, respectively, and total CPU usage by 10.69% compared with kernel TCP. In training, ByteExpress reduces P99 model parameter push latency by up to 74.0% compared with TCP, with no PFC pauses observed at runtime.
| Comments: | 14 pages, 16 figures |
| Subjects: | Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC); Networking and Internet Architecture (cs.NI); Systems and Control (eess.SY) |
| Cite as: | arXiv:2610.12815 [cs.LG] |
| (or arXiv:2610.12815v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12815 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yibo Huang [view email]
[v1]
Thu, 8 Oct 2026 23:57:00 UTC (657 KB)
来源：arXiv：cs.LG · arxiv.org