Bigger batch means more parallelism means higher throughput. The logic is sound for compute-bound work — and LLM decoding is not compute-bound.
What going from batch 8 to 32 actually does
- —Throughput rises by roughly 40% — far less than the 4× the batch size suggests.
- —p95 latency can go from around 150ms to 450ms, three times worse.
- —GPU utilisation still reads 85%+, so the dashboard says everything is fine.
Why: decoding is memory-bound
In a decode step, the overwhelming majority of time goes to moving weights across the memory bus, not to arithmetic. Roughly speaking, weight loading dominates, KV-cache reads take a slice, and actual compute is a small remainder.
Adding requests to the batch fills compute capacity you were never short of, while every request now waits behind a longer queue. That is the whole trap: you buy a resource you had spare and pay for it in the resource users feel.
Where throughput actually comes from
01Continuous batchingAdmit and retire requests independently instead of waiting for a whole batch to finish.
02KV-cache reuseAcross turns in a conversation, so the prefix is not recomputed every time.
03QuantisationFewer bytes per weight means less data crossing the bus per step — attacking the real bottleneck.
Tune batch size against a latency budget rather than against a throughput target, and keep the smallest batch that meets your throughput need.
