This week’s Paper Club is all about inference.

For every FLOP spent training a model, roughly six go to serving it each year — and getting that part right has produced over $220 billion in market cap in the last decade.

But inference is much more than calling model.forward. Every token has to move through a complicated stack of routing, caching, scheduling, kernels, memory, and hardware, all while balancing speed, cost, and quality.

We brought together researchers and builders from @togethercompute, @ollama, @luminal_ai, and @sgl_project & @radixark to take us through that stack from end to end. We’ll see how a token gets from an API request to a GPU and back, why the same open model can perform differently depending on who serves it, and how new approaches to compilers and caching are making inference faster and more efficient.

00:00 - Why inference matters more than training
08:24 - Three kinds of inference innovation
12:14 - Dan Fu (@realDanFu): the lifecycle of an inference request
42:18 - Jeff Morgan (@jmorgan): not all open model tokens are the same
1:04:31 - Joe Fioti (@joefioti): compiling models with search, not heuristics
1:30:30 - Liangsheng Yin (@lsyincs): SGLang and memory for the agent era