Rohan Paul· @rohanpaul_ai · X·· 3 小时前AI 评分26
AI 导读
Inferact 的 Yifan Qiao 在 DigitalOcean Open Intelligence Summit 的 Serve 分场分享如何在智能体生产负载上运行 vLLM。
正文
Agent traffic is a strange workload for an inference server.
Each turn resends a long, nearly identical prefix (system prompt, tool schemas, history) and asks for a short tool call back.
At DigitalOcean’s Open Intelligence Summit, Yifan Qiao of Inferact is giving a Serve track session on running vLLM on agentic production workloads.
That shape of traffic is why prefix caching and KV-cache reuse matter more for agents than raw tokens per second. A cache miss on a long prefix means paying for the full prefill again, on every turn of a long task.
Who gets to shape the future of AI? Today we're bringing together some of the biggest voices in open models, data, and agents to answer that question, including @rbren_dev of @OpenHandsDev, @zlxi02 of @inferact, @GerryFPatia of @boltdotnew, @vinaygo of @Amplitude_HQ and @latkins of @arcee_ai. The Open Intelligence Summit, co-hosted with @nvidia. Watch live, starting at 1 PM PT: Keynote: http://youtube.com/live/IhWSgZlVizQ Build: http://youtube.com/live/eN0TqmeBNhw Serve: http://youtube.com/live/fkT-MhcAm-4 Run: http://youtube.com/live/QMrY1xZ-xGg在 X 查看被引用的帖子
来源：Rohan Paul · x.com