Latency

The Streaming article introduced Time To First Token as a real, named metric. This article gives that concept, and the broader measurement it belongs to, full treatment: latency.

The simple definition

Latency is the time delay between a request being made and a response beginning to arrive. Recall from the Streaming article’s core distinction — streaming doesn’t change total generation time, only when the user starts seeing output. Latency is precisely the measurement of that waiting period, and it comes in a few genuinely distinct, specifically named forms that matter for different reasons.

Why latency has to be broken into specific, separate measurements

Recall from the Autoregressive Generation article that a model generates tokens one at a time, in sequence. This means “how long did the user wait” is actually several different, meaningful numbers, not one. Time To First Token (TTFT) measures how long until the very first piece of output appears — the number streaming, covered in the previous article, is specifically designed to minimize.

Inter-token latency measures the gap between each subsequent token once generation has started. Total latency measures the complete time until the entire response finishes. A system can have excellent TTFT but slow inter-token latency, or vice versa — genuinely different bottlenecks, requiring genuinely different fixes.

flowchart LR
    A[Request sent] --> B["TTFT: time until first token appears"]
    B --> C["Inter-token latency: gap between each following token"]
    C --> D["Total latency: time until the entire response is complete"]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of ordering food at a restaurant — how long until your waiter first acknowledges your order (TTFT), how long between each course actually arriving at your table (inter-token latency), and how long until your entire meal is finished being served (total latency). A restaurant could be excellent at one of these and genuinely poor at another, and a diner’s overall satisfaction depends on all three in different ways.

Where this breaks down: A restaurant’s pacing involves kitchen staff making real, judgment-based decisions about timing. A model’s latency at every stage is a precise, measurable consequence of its architecture, hardware, and serving software, covered throughout this phase — the exact computational cost of each forward pass, not a chef’s deliberate pacing choices.

The real, viral story: a company that made speed itself the entire product

This deserves to be told in full, because it’s a genuine, dramatic, real-world demonstration of how much latency alone can matter as a competitive advantage.

In February 2024, a company called Groq went viral after public demonstrations showed its custom-built chip — which Groq calls an LPU, or Language Processing Unit, deliberately not a GPU — generating around 500 tokens per second on the Mixtral model, compared to roughly 40 tokens per second for the publicly available ChatGPT-3.5 at the time, a genuine, measured tenfold difference.

Groq was founded by Jonathan Ross, one of the original architects of Google’s TPU, covered in its own article earlier in this phase, and built its entire chip architecture around minimizing exactly the kind of latency this article describes, rather than optimizing for raw parallel throughput the way a conventional GPU does.

Groq’s own CEO cited real, published research that improving website response time by just 100 milliseconds increases user engagement by roughly 8 percent on desktop and 34 percent on mobile — a genuine, concrete illustration of why latency, not just raw model capability, has become a real, separate axis of competition across the industry.

A concrete example, layered

For a simple beginner example: a voice assistant needs extremely low latency specifically at the TTFT stage, since even a one-second pause before the assistant starts responding feels awkward and unnatural in real, spoken conversation, in a way a similar pause feels less jarring in text chat. For a production example: real, published benchmarking platforms like ArtificialAnalysis.ai directly measure and compare TTFT and total latency across providers including OpenAI, Anthropic, and Groq, and these published numbers directly inform which provider a real company chooses for latency-sensitive applications like live customer support or real-time translation.

Why lower latency almost always costs something real

It’s worth being honest about a genuine trade-off here, not presenting minimal latency as a costless goal. Groq’s hardware, referenced above, is a real, specialized, and expensive chip built specifically to minimize latency, at the cost of the general-purpose flexibility a standard GPU provides — recall from the GPU article that specialization always trades away flexibility for a specific advantage. Lower latency, achieved through better hardware, smaller models, or more aggressive quantization (covered later in this phase), typically comes at some real cost — in dollars, model capability, or engineering complexity — rather than being a pure, unconditional win.

The real numbers that made latency a boardroom concern, long before AI

This is worth grounding in genuine, widely cited data, since concern over latency’s business impact predates AI chatbots by nearly two decades. In a real, published 2006 experiment, Amazon found that deliberately adding just 100 milliseconds of delay to its website reduced sales by 1 percent — a real, measured, and now-famous result.

Google found something structurally similar around the same time: an additional 400 milliseconds of delay in returning search results reduced daily search volume by roughly 8 million searches.

These weren’t AI-specific findings at all, but they directly explain why the latency-obsessed engineering culture this entire phase describes — Groq’s specialized chips, streaming’s perceived-speed trick, vLLM’s memory optimizations — exists in the first place: the underlying business case for minimizing latency was established well before language models made it newly relevant again.

Common misconception

Diagnose where the waiting happened

Suppose total latency is 5 seconds. A trace might divide it like this:

Part	Time
Network and gateway	0.2 s
Queue waiting	1.3 s
Prompt prefill	0.5 s
Token generation	2.4 s
Tool call	0.6 s
Total	5.0 s

The correct optimization depends on the largest part. A faster GPU may reduce prefill or generation time, but it will not fix a slow external search tool. More replicas may reduce queue time, but they will not shorten an unnecessarily long output.

Keep historical benchmarks labeled

The Groq, Mixtral, ChatGPT-3.5, Amazon, and Google numbers above describe tests from their stated periods. They remain useful examples of why latency matters, but they are not current rankings of today’s GPT, Gemini, Claude, or Groq services.

Current comparisons must be measured again with the same prompt, output length, concurrency, region, service tier, and date.

Break one response into waiting times

Suppose an answer begins after 0.8 seconds and finishes after 4.8 seconds:

0.0 s  request sent
0.2 s  network, queue, and setup completed
0.8 s  first visible token arrives       <- TTFT
4.8 s  final token arrives               <- end-to-end latency

• Time to First Token (TTFT) is the wait before generation becomes visible.
• Inter-token latency is the time between generated token pieces.
• End-to-end latency is the complete wait from request to finished response.

For this example, the model streams for 4 seconds after the first token. If it produces 200 output tokens during those 4 seconds, its approximate generation rate is 50 output tokens per second.

What can increase latency?

network + queue + prompt processing + token generation + tool calls + safety checks

A longer input increases prompt-processing work. A longer output increases generation time. Heavy traffic increases queue time. Tool-using agents may wait for searches, databases, code, or other models.

This is why comparing “model latency” requires the same region, prompt, output length, concurrency, streaming definition, and measurement method.

Production practice

Teams monitor percentiles such as p50, p95, and p99, not only an average. If p95 TTFT is 2 seconds, 95% of measured requests began responding within 2 seconds, while 5% took longer.

Verified sources

• NVIDIA GenAI-Perf measurement concepts
• vLLM benchmarking documentation

A frequent beginner assumption: that “latency” and “speed” refer to the exact same single thing an AI product can simply be good or bad at. As this article’s TTFT-versus-inter-token distinction explained, latency actually breaks into genuinely separate, independently measurable stages, and a real system can excel at one while struggling with another — which is exactly why serious benchmarking, like the ArtificialAnalysis.ai comparisons referenced above, reports these figures separately rather than collapsing them into one single “speed” number.

Where this fits in what comes next

You now understand latency as a precise, multi-part measurement of response delay. The next article, Throughput, covers the complementary metric this phase has referenced throughout — not how fast one user’s response arrives, but how much total work a system can handle across many users at once.

In one sentence

Latency measures the delay between a request and a response, breaking into genuinely distinct stages — time to first token, inter-token latency, total latency — and Groq’s real, viral 2024 demonstration of roughly 10x faster token generation than contemporary GPU-based systems proved, concretely and publicly, just how much of a genuine competitive advantage minimizing this specific metric can become.

Related Terms

• Streaming
• Autoregressive Generation
• Throughput
• Tokens Per Second (TPS)
• GPU