Does continuous batching hurt latency for a single request?

Not appreciably: each request keeps being processed token by token in its own slot. The gain shows up in the system's overall throughput, how many requests get served in parallel, not in the speed of any single response.

What is continuous batching in LLM serving?

LLM serving technique that frees a finished request's slot right away, without waiting for the others in queue.

Continuous batching (or in-flight batching) is the scheduling technique an LLM inference engine uses to group multiple requests on the same hardware while freeing each one's slot as soon as it finishes generating its response, instead of waiting for every request in the original group to finish. In static batching, the earlier approach, a fixed group of requests is processed together and the slot stays occupied until all of them complete, even when one request is much shorter than the others: that capacity sits wasted while waiting, serving no one. Here the term "batching" refers to scheduling inference requests in production, a problem distinct from batch processing in data ETL, which describes offline data processing in scheduled chunks: two concepts sharing a name but in completely different domains. Continuous batching originates with the Orca paper (OSDI 2022) and was made mainstream by vLLM in 2023 and then by DeepSpeed-FastGen in early 2024, and is today the implicit standard of nearly every self-hosted serving engine.

How iteration-level scheduling works

The engine recomputes the batch's composition at every single decoding step (token by token), not once at the start. As soon as a sequence hits its end-of-response token, its slot is immediately reassigned to a queued request, without waiting for the rest of the group. This only works if each request's KV cache memory can be allocated and freed dynamically instead of reserved upfront in one block: that is exactly what PagedAttention does, managing that memory in pages assigned just as they are needed, with waste under 4% as reported in the vLLM paper versus the tens of percentage points typical of static allocation, which is why vLLM made continuous batching practical at scale.

Why throughput rises without changing hardware

The gain does not come from more powerful GPUs but from less idle GPU time: in static batching, every free slot while waiting for longer requests is paid-for compute capacity going unused. Continuous batching fills those gaps in real time, which translates into more requests served per second on the same graphics card. The DeepSpeed-FastGen paper and the literature following Orca document order-of-magnitude throughput gains over static batching under equivalent latency conditions, and vLLM has made this behavior the de facto standard of the open-source ecosystem.

An enterprise example

A customer support service answering both "what is my balance" (a few words) and "summarize this ten-page contract and flag risky clauses" (thousands of tokens) is the textbook case. Under static batching, every short request would stay locked in the same slot until even the longest request in the group finishes, wasting GPU capacity for the entire duration difference. Under continuous batching, the short request frees its slot within seconds and that slot immediately goes to the next user in queue, while the long request keeps running on its own slot independently of the others.

Why it matters for decision makers

For a company evaluating a self-hosted inference engine, continuous batching is often the single optimization with the best cost-to-benefit ratio: it requires no additional hardware and no different model, and in current practice vLLM, TensorRT-LLM (which calls it "in-flight batching") and SGLang implement it by default. The question worth asking when choosing is not whether a platform supports it, but how well it handles cases with highly variable response lengths, which is exactly the scenario where this technique makes the biggest economic difference.