arXiv：cs.AI· Shinan Liu, Ted Shaowang, Gerry Wan, Jeewon Chae, Jonatas Marques, Sanjay Krishnan, Nick Feamster·· 5 小时前
FlowWise：面向流式流量的有状态快慢模型服务系统
FlowWise: Stateful Fast-Slow Model Serving for Streaming Traffic Intelligence
AI 导读
FlowWise 是一个面向流式、有状态流量智能的有状态快慢模型服务系统，采用不确定性驱动的快慢级联：先用单包模型给出早期预测，仅将不确定的流升级到等待更多数据包的慢模型。系统在 16 ms 内服务 76.3% 的流，中位端到端延迟改善 40.5x；单台 16 核 CPU 服务器每秒可承载超 48.5k 新流，FlowWise 已开源，论文被 SoCC 2026 接收。
正文
Abstract:Network traffic analysis in cloud and internet service provider (ISP) networks increasingly relies on machine learning (ML) models, as encryption and service consolidation make rule-based classification unreliable. At high flow-arrival rates, however, new flows can arrive faster than end-to-end analysis pipelines can process. Unlike typical cloud inference workloads, packets from the same flow arrive over time and require per-flow state. This streaming, stateful setting limits naive scale-out with stateless workers. This paper presents FlowWise, a serving system for streaming, stateful flow intelligence on commodity CPU/GPU servers that jointly selects (i) how many packets to collect and (ii) which model(s) to apply for each flow to balance latency, service rate, and accuracy. Across three representative tasks, inference time across candidate models differs by 1.8x-141.3x, while packet waiting time can exceed feature computation and inference by 6-8 orders of magnitude at the tail. FlowWise exploits these gaps with an uncertainty-driven fast-slow cascade: it issues early predictions from a one-packet model and escalates only uncertain flows to slower models that wait for additional packets and use richer features. A queue-centric runtime maintains per-flow state, performs asynchronous feature joining, and uses brokered, flow-ID-keyed streams to distribute workload across heterogeneous workers. FlowWise serves 76.3% of flows within 16 ms, improving median end-to-end serving latency by 40.5x while increasing service rate and maintaining similar accuracy to always running the slow model. With thousands of packet-header features per flow, our scale-up benchmark sustains over 48.5k new flows per second on a single 16-core commodity CPU server, comparable to reported flow-arrival rates in city-backbone traces. FlowWise is open sourced at this https URL.
| Comments: | Accepted by SoCC 2026 |
| Subjects: | Networking and Internet Architecture (cs.NI); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2402.03694 [cs.NI] |
| (or arXiv:2402.03694v3 [cs.NI] for this version) | |
| https://doi.org/10.48550/arXiv.2402.03694 arXiv-issued DOI via DataCite |
Submission history
From: Shinan Liu [view email]
[v1]
Tue, 6 Feb 2024 04:28:33 UTC (2,552 KB)
[v2]
Thu, 24 Oct 2024 14:15:42 UTC (3,473 KB)
[v3]
Sun, 11 Oct 2026 00:38:29 UTC (1,621 KB)
来源：arXiv：cs.AI · arxiv.org