Throughput

The Latency article covered how fast one individual request feels. This article covers the complementary, and often competing, metric: throughput.

The simple definition

Throughput measures how much total work a system can process in a given amount of time — commonly measured as total tokens generated per second across all users combined, not any single user’s individual experience. Recall from the Batch Inference article’s core motivation — grouping requests together to use GPU capacity more efficiently. Throughput is precisely the metric that improves when batching works well: the same hardware serving more total requests, or generating more total tokens, in the same amount of time.

Why latency and throughput genuinely pull in opposite directions

This is worth being direct about, since it’s the central tension underlying nearly every serving decision covered throughout this phase. Recall from the Batch Inference article that grouping requests together improves overall efficiency, but makes any single request wait longer for the whole batch to finish — better throughput, worse latency for that individual request.

Recall from the Latency article’s Groq example that a system optimized specifically to minimize one user’s wait time often can’t simultaneously maximize how many total users it serves. A system genuinely has to choose where on this spectrum to sit, since maximizing both simultaneously, for every possible workload, isn’t actually achievable.

flowchart LR
    A[Optimize for Latency] --> B[Fast for one user, less total capacity]
    C[Optimize for Throughput] --> D[More total capacity, slower for any one user]
    B -.genuine trade-off.-> D

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a highway system — a single, dedicated express lane gets one car to its destination as fast as physically possible (latency), while a highway with many regular lanes moves a much larger total number of cars per hour, even though any individual car in regular traffic takes somewhat longer than it would in that dedicated express lane (throughput). Building for one genuinely limits how well you can serve the other with the same total road space.

Where this breaks down: A highway’s lanes are a fixed, physical allocation, decided once during construction. An inference server’s throughput-versus-latency balance, as covered in the Inference Server article’s PagedAttention example, can be tuned dynamically and continuously by real, automated scheduling software, adjusting batch sizes and request handling in real time based on current traffic, rather than being locked into one fixed physical design.

The real numbers behind why this trade-off matters so much economically

Recall directly from the Inference Server article’s vLLM story. The real, published PagedAttention research specifically targeted throughput — achieving 2 to 4 times more total tokens processed per second on identical hardware, letting LMSYS cut its GPU count in half while serving more total traffic. This is worth being explicit about: that improvement was a genuine throughput win, not primarily a latency one — any individual user’s response wasn’t necessarily faster, but the total system could serve dramatically more users on the same hardware, directly reducing real, measured infrastructure cost.

A concrete example, layered

For a simple beginner example: a customer support chatbot serving thousands of simultaneous users cares primarily about throughput — total tokens processed per second across everyone — since the company’s real infrastructure cost scales with total capacity needed, not with how instantaneously any one specific conversation responds.

For a production example: cloud AI providers like OpenAI and Anthropic publish separate, distinct pricing and performance tiers precisely because throughput-optimized and latency-optimized serving represent genuinely different, real engineering configurations — a batch-oriented, high-throughput workload and a real-time, low-latency chat application are treated as different problems requiring different infrastructure choices, not the same service simply used two different ways.

Why a company has to actively choose which metric to prioritize

It’s worth being direct about the real, practical decision this creates for any team deploying a model. A live, interactive chat application should generally prioritize low latency, even if that costs some overall throughput, since a slow-feeling individual response directly damages user experience, as covered in the Latency article’s real engagement statistics. A backend system processing millions of documents overnight, echoing the Batch Inference article’s core use case, should prioritize throughput instead, since nobody is watching any single document’s processing time in real time.

Common misconception

Concurrency and saturation

Concurrency is the number of requests being processed or waiting at the same time. Increasing concurrency can help the server form efficient batches and raise throughput.

That improvement has a limit:

few requests      -> hardware partly idle
more requests     -> better batches and higher throughput
too many requests -> full hardware, growing queue, sharply higher latency

The point where adding work no longer produces a similar increase in completed work is called saturation. Beyond that point, extra requests mostly wait.

One model-serving example

For a Llama deployment on vLLM or TensorRT-LLM, an engineer can raise the number of simultaneous requests and record both total output-token throughput and p95 latency. The best operating point is not necessarily the maximum possible throughput; it is the highest throughput that still meets the application’s latency target and memory limits.

This experiment must name the Llama version, hardware, input length, output length, numerical format, and serving settings. Otherwise, the throughput number cannot be reproduced or compared fairly.

Put numbers on throughput

Suppose one server completes 240 requests in 60 seconds:

request throughput = 240 / 60 = 4 requests per second

If those requests produce 30,000 output tokens altogether:

output-token throughput = 30,000 / 60 = 500 output tokens per second

Both numbers describe the same minute, but they answer different questions. Request throughput is useful for capacity planning. Token throughput reflects how much generated text the system produced.

Throughput is not one user’s speed

An inference server might produce 5,000 tokens per second across 100 active users while each user receives only about 50 tokens per second. System throughput and per-user generation speed must therefore be reported separately.

Total server throughput
        = work delivered across all active requests

Per-user TPS
        = generation rate observed by one request

A real serving connection

vLLM’s PagedAttention work reported 2–4 times higher throughput than the compared serving systems on evaluated workloads. NVIDIA TensorRT-LLM combines in-flight batching, paged KV caching, optimized kernels, and lower-precision formats to pursue the same production goal: more useful work from the available GPUs while meeting latency targets.

Verified sources

• vLLM project and PagedAttention paper
• NVIDIA TensorRT-LLM documentation

A frequent beginner assumption: that a “faster” AI system is simply faster in every sense — quick for individual users and capable of handling more total volume, both at once, with no trade-off involved. As this article’s highway analogy and the real vLLM numbers demonstrated, this isn’t how serving infrastructure actually works — genuine engineering choices trade one of these metrics against the other, and a system’s real speed has to be understood as at least two separate, sometimes competing numbers, not one single measure of “fast.”

Where this fits in what comes next

You now understand both halves of the fundamental latency-versus-throughput trade-off. The next article, Tokens Per Second (TPS), covers the single, specific, most commonly cited number that actually combines aspects of both — the metric behind Groq’s real, viral 500-tokens-per-second demonstration covered in the previous article.

In one sentence

Throughput measures a system’s total processing capacity across all users combined, and it exists in genuine, direct tension with latency — improving one typically costs something on the other — which is exactly why the real, published vLLM research achieved its dramatic gains specifically by targeting throughput, letting real companies serve far more traffic on the same hardware rather than making any single response faster.

Related Terms

• Latency
• Batch Inference
• Inference Server
• Tokens Per Second (TPS)
• GPU