Inference – LLMs | TechByteByByte

Before you continue: three tools for this module

Inference: using fixed learned parameters to produce output.
Logit: a raw score for a possible next token.
Decoding: the rule that chooses tokens from model scores.

You do not need to memorize these yet. Use this map when the terms reappear.

Begin with the central question

What hidden problem does Inference solve inside a real language-model system?

Keep that central question about Inference in mind. The definitions, numbers, diagrams, and code examples below answer it one piece at a time.

trained parameters + prompt → forward passes → generated response

1. What You Will Learn

Learning outcomes

Define inference and distinguish it from training.
Separate prompt prefill from token-by-token decoding.
Trace latency, throughput, memory, and KV-cache effects.
Explain what an inference server contributes beyond the model itself.

In one sentence

💡 Big picture

Inference is the stage where a trained model uses fixed learned parameters to answer a new prompt.

2. Why This Module Exists

The problem this module solves

Training teaches the model; inference uses what was learned.
Understanding prefill, decoding, caching, and serving explains why responses have different speeds and costs.

3. Intuition

processing your prompt for the first time (prefill) is a very different computational problem from generating each subsequent token (decode). Prefill processes many tokens in parallel, once. Decode processes one new token at a time, repeatedly — and without caching, it would redundantly reprocess the entire growing sequence at every single step.

4. Core Concept

Prefill:    the INITIAL forward pass over the ENTIRE prompt, all
            at once -- highly parallelizable, computed ONCE per
            request

Decode:       GENERATING new tokens ONE AT A TIME (Module 7's
            loop) -- each step depends on the previous step's
            output, inherently SEQUENTIAL

KV cache:       stored Key and Value vectors (Transformers course)
            from all previously-processed tokens, reused instead
            of recomputed at every decode step

Prompt
 ↓
Prefill                (process the WHOLE prompt, populate the
                       KV cache)
 ↓
KV Cache
 ↓
Generate token            (decode step 1 -- uses cached K/V +
                          the new token's Q)
 ↓
Update cache
 ↓
Generate next token          (decode step 2 -- reuses cache again)
 ↓
...

5. How It Works — Step by Step

1. PREFILL: the entire prompt is processed in ONE forward pass --
   computing Key and Value vectors for EVERY prompt token, at
   EVERY layer, and STORING them in the KV cache
2. The FIRST generated token comes from this prefill pass's final
   position's logits (Module 5)
3. DECODE begins: for each new token, compute Q, K, V for ONLY
   the newest token
4. Retrieve ALL previously-cached K, V (no recomputation) and
   combine with the new token's K, V for the attention
   computation
5. Generate the next token, APPEND its K, V to the cache, repeat
6. WITHOUT this cache, every decode step would need to recompute
   K, V for the ENTIRE sequence so far -- exactly the wasteful
   pattern demonstrated in the Transformers course, quantified
   again below specifically for LLM serving

6. Mathematical Intuition

Read the mathematics as a story

trained parameters + prompt → forward passes → generated response

First locate the input, operation, and output. Then treat the formula as a compact description of that journey rather than a collection of symbols to memorize.

Prefill cost scales quadratically with prompt length (attention’s O(n²) cost, Transformers course) — verified directly below.

Decode cost, per step, is O(current_sequence_length) without caching (recomputing everything) but only O(1) new computation per step with caching (only the newest token’s Q/K/V, attending to the already-cached rest) — the same quadratic-vs-linear distinction from the Transformers course, now quantified specifically for a realistic serving scenario.

Analogy: The Restaurant Cooking Pipeline & Pre-chopped Bowls Think of processing prompt text and generating tokens in terms of restaurant logistics:

Prefill Phase (The Prep Work): When a table of 8 orders, the chef chops all onions, carrots, and herbs in parallel (the prompt tokens). This takes some effort but is done once per party. The chopped ingredients are placed in bowls on the counter (stored in the KV cache).

Decode Phase (Plating Garnish): The chef plates the dishes one by one, adding a final token garnish.

Without Caching (Wasteful Chef): For plate 2, the chef throws away all unused prep, reads the recipe, and chops all onions and carrots from scratch again. For plate 3, they chop them all again.

With Caching (Efficient Chef): The chef simply scoops from the pre-chopped bowls on the counter (reusing the KV cache), chops only the single new garnish item, and finishes plating instantly.

📊 Visual Flowchart: Prefill vs. Decode Caching Lifecycle

Here is how K and V tensors are calculated once and retrieved dynamically at subsequent steps:

graph TD
    classDef prefill fill:#3498db,stroke:#333,stroke-width:1px,color:#fff;
    classDef decode fill:#2ecc71,stroke:#333,stroke-width:1px,color:#fff;
    classDef cache fill:#f1c40f,stroke:#333,stroke-width:1px,color:#fff;

    subgraph Phase1 ["1. Prefill Phase (Prompt Execution)"]
        Prompt["Input Prompt (100 tokens)"] --> ComputeAll["Parallel forward pass: compute Q, K, V for all 100 tokens"]:::prefill
        ComputeAll --> WriteCache["Write K, V to GPU Memory Cache"]:::cache
    end

    subgraph Phase2 ["2. Decode Phase (Token Generation Steps)"]
        WriteCache --> NextStep["Generate Token step i"]
        NextStep --> NewProj["Compute Q, K, V for the ONE newest token ONLY"]:::decode
        ReadCache["Read previous tokens' K, V from Cache"]:::cache --> Attention["Attention: Q_new @ [K_cached + K_new]"]
        NewProj --> Attention
        NewProj --> UpdateCache["Append newest K, V to Cache"]:::cache
        Attention --> GenWord["Output next token logits"]:::decode
        GenWord --> Loop["Loop back for next step"]
    end

7. Small Worked Example

Walk through the example

Name what each input represents.
Follow one transformation at a time.
Translate the result back into ordinary language.

The purpose is to reveal the mechanism, not merely display an answer.

For a 100-token prompt generating 50 new tokens: without a KV cache, each of the 50 decode steps redundantly recomputes attention over the entire growing sequence (100, then 101, then 102… tokens) from scratch. With caching, each step only computes the newest token’s contribution, reusing everything already computed.

8. Python Example

What the code will demonstrate

The code builds a tiny version of the mechanism, prints values you can inspect, and connects them to the worked example. Predict the direction of the result before running it.

Python symbols used below

NumPy (np) stores numeric vectors and matrices.
np.array(...) creates a numeric collection.
Library calls perform the same conceptual steps shown above at a larger scale.

# Build a small, inspectable example of Inference.
# Follow the inputs, transformations, and output in order.
d_model = 64
seq_lens = [50, 100, 200, 400, 800]

def attention_flops(seq_len, d_model):
    return 2 * seq_len * seq_len * d_model   # core QK^T + softmax@V cost

print("PREFILL cost -- scales QUADRATICALLY:")
for n in seq_lens:
    print(f"  Prompt length {n:4d} -> ~{attention_flops(n, d_model):,} FLOPs")

print(f"\nDoubling 400->800: FLOPs multiplier = {attention_flops(800,d_model)/attention_flops(400,d_model):.1f}x")

# --- DECODE: WITHOUT vs WITH KV cache ---
def decode_flops_no_cache(prompt_len, num_new_tokens, d_model):
    total = 0
    for i in range(num_new_tokens):
        current_len = prompt_len + i + 1
        total += attention_flops(current_len, d_model)   # recomputes EVERYTHING each step
    return total

def decode_flops_with_cache(prompt_len, num_new_tokens, d_model):
    total = 0
    for i in range(num_new_tokens):
        current_len = prompt_len + i + 1
        total += 2 * current_len * d_model   # only new token's Q against cached K
    return total

prompt_len = 100
num_new_tokens = 50
no_cache = decode_flops_no_cache(prompt_len, num_new_tokens, d_model)
with_cache = decode_flops_with_cache(prompt_len, num_new_tokens, d_model)

print(f"\nPrompt length: {prompt_len}, generating {num_new_tokens} new tokens")
print(f"  WITHOUT KV cache: {no_cache:,} total FLOPs")
print(f"  WITH KV cache:    {with_cache:,} total FLOPs")
print(f"  Reduction factor: {no_cache/with_cache:.1f}x")

Expected Output:

PREFILL cost -- scales QUADRATICALLY:
  Prompt length   50 -> ~320,000 FLOPs
  Prompt length  100 -> ~1,280,000 FLOPs
  Prompt length  200 -> ~5,120,000 FLOPs
  Prompt length  400 -> ~20,480,000 FLOPs
  Prompt length  800 -> ~81,920,000 FLOPs

Doubling 400->800: FLOPs multiplier = 4.0x

Prompt length: 100, generating 50 new tokens
  WITHOUT KV cache: 102,134,400 total FLOPs
  WITH KV cache:    803,200 total FLOPs
  Reduction factor: 127.2x

9. How It Works

Prefill confirms the Transformers course’s quadratic scaling directly: doubling prompt length from 400 to 800 produces exactly a 4.0x FLOPs increase.
Decode without caching costs 102,134,400 FLOPs to generate just 50 tokens after a 100-token prompt — because every single step redundantly recomputes attention over the entire, ever-growing sequence.
Decode with caching costs only 803,200 FLOPs for the exact same 50 tokens — a 127.2x reduction. This dramatic gap is why the KV cache isn’t a minor optimization; it’s what makes serving real, multi-token generation requests economically and practically feasible at all.

10. Why Generation Is Expensive

1. Prefill's quadratic cost with prompt length (verified directly)
2. Decode's inherently SEQUENTIAL nature (Module 7) -- each step
   must wait for the previous one, limiting parallelization within
   a single request
3. EVEN WITH caching, growing KV cache consumes real, growing GPU
   memory (Transformers course) as context length increases

11. Latency, Throughput, and Batch Inference

Latency:            time to get a response for ONE request

Throughput:            total tokens generated across ALL
                    concurrent requests, per unit time

Batch inference:         processing MULTIPLE requests together,
                      exploiting shared GPU compute more
                      efficiently -- improves throughput, can
                      affect individual-request latency

These are genuinely distinct, sometimes competing, production metrics — optimizing purely for one can hurt the other (Module 24 covers specific techniques like continuous batching that help balance both).

12. How Is This Used in Modern AI?

Trace it through a real model call

user message → assembled context → LLM computation → decoded output → application checks

This topic affects one stage of that path; it is not the complete product. Hosted GPT- and Gemini-style applications also add instructions, safety systems, retrieval, tools, serving infrastructure, and evaluation around the model.

🤖 How Is This Used in Modern AI?

Every production LLM serving system implements prefill/decode as structurally distinct phases with KV caching as standard, essential infrastructure — not an optional feature. The dramatic reduction verified directly (127x for a modest example) is precisely why no serious production system operates without it.

13. How Is This Used in Agentic AI?

Separate the model from the runtime

goal + state + tool results → LLM proposal → runtime validation → execution or response

The LLM proposes text or a structured action. Ordinary application code controls permissions, tools, retries, memory, and execution.

Direct relevance to Agentic AI: Very High. Long agent conversations with growing context (system prompt, history, tool results) directly stress both prefill cost (reprocessing a growing prompt on each new turn, unless the KV cache can be reused across turns) and the KV cache’s memory footprint.

Understanding this is essential for reasoning about why long-running agent sessions have real, growing latency and cost characteristics.

14. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: treating prefill and decode as the same computational problem.

Why it is incorrect: As demonstrated, they have fundamentally different cost structures — prefill is a one-time, highly parallel, quadratic-cost operation; decode is repeated, sequential, and (with caching) incrementally linear-cost.

⚠️ Mistake

Incorrect idea: assuming KV caching is a minor optimization.

Why it is incorrect: As verified directly with a 127x reduction, it’s the difference between a practically-servable system and a computationally catastrophic one.

⚠️ Mistake

Incorrect idea: conflating latency and throughput.

Why it is incorrect: They’re genuinely different metrics that can trade off against each other — a system optimized purely for throughput (via aggressive batching) may show worse individual-request latency.

15. Important Distinctions

Prefill	Decode
Processes the ENTIRE prompt at once	Generates ONE token at a time
Highly parallelizable	Inherently sequential
Quadratic cost with prompt length — verified directly	Linear cost per NEW token, WITH caching — verified directly

Latency	Throughput
Time for ONE request	Total tokens across ALL concurrent requests, per unit time

16. When to Use

KV caching should always be used in any real LLM serving deployment — there’s no legitimate reason to omit it given the dramatic, verified cost difference. Batch inference should be used when throughput across many concurrent requests matters more than any single request’s absolute minimum latency.

17. When Not to Use

Not applicable for KV caching — it’s standard, essential infrastructure. Heavy batching may be inappropriate for latency-critical, single-user interactive applications where consistent, low per-request latency matters more than aggregate throughput.

18. Production Considerations

KV cache memory grows with context length — a genuine, real constraint on how many concurrent requests (and how long a context) a given amount of GPU memory can support (Module 24 covers techniques like GQA/MQA that reduce this further, per the Transformers course).
Prefill and decode often have different optimal batching strategies in real serving systems, given their different computational characteristics.
Long agent conversations’ repeated prefill cost (if context can’t be cached across turns) is a genuine, practical latency and cost concern worth architecting around.

19. What You Should Remember

Prefill processes the whole prompt once, with quadratic cost in prompt length — verified directly (4x cost for 2x length).
Decode generates tokens one at a time, sequentially — verified directly to cost 127x more without KV caching than with it, for a realistic example.
Latency and throughput are distinct, sometimes competing production metrics — batch inference trades between them.

20. Interview Questions

Beginner

Q: What is the difference between the prefill and decode phases of LLM inference?

Ans: Prefill is the initial processing of the entire prompt in one parallel forward pass, computing and caching Key/Value vectors for every prompt token.

Decode is the subsequent, sequential generation of new tokens one at a time, each step reusing the cached information from prefill (and previous decode steps) rather than reprocessing everything from scratch.

Intermediate

Q: Why does KV caching matter so much for decode-phase efficiency?

Ans: Without caching, generating each new token would require recomputing attention over the entire, ever-growing sequence from scratch — a massively redundant computation, since earlier tokens’ Key and Value vectors never change once computed (a direct consequence of causal masking, Module 11).

Verified directly in this module: caching reduced total decode FLOPs by over 127x for a realistic 50-token generation scenario, turning an increasingly expensive quadratic-per-step cost into a much cheaper, roughly linear one.

Advanced

Q: Why does prefill cost scale quadratically with prompt length while KV-cached decode cost scales roughly linearly per new token?

Ans: Prefill computes the full attention score matrix across all n prompt tokens at once — an n × n computation, hence quadratic in prompt length, verified directly (doubling length produced a 4x FLOPs increase).

Decode, with caching, only computes the NEW token’s Query against the already-cached Keys of all previous tokens — a computation proportional to the current sequence length, not squared, since the n × n full-matrix computation isn’t repeated; only one new row is added each step. This structural difference is exactly why the two phases have such different serving cost profiles.

Scenario

**Q: A team’s LLM-powered application shows high, growing latency specifically for very long conversations, even though individual messages are short.

Using this module, what would you investigate?** A: I’d investigate two things directly connected to this module: first, whether the KV cache is being reused efficiently across conversation turns, or whether the system is unnecessarily reprocessing (re-prefilling) the growing conversation history on every turn — a genuine, avoidable cost.

Second, I’d check whether growing KV cache memory usage (which scales with total context length) is causing memory pressure or forcing less efficient batching as the conversation grows — both are real, expected consequences of longer contexts under this module’s cost model, not signs of an unrelated bug.

AI Engineering

Q: Why do production LLM serving systems treat prefill and decode as architecturally distinct phases, sometimes even serving them on different infrastructure?

Ans: Because they have genuinely different computational characteristics and optimization needs — prefill is a one-time, highly parallel, compute-intensive operation per request, while decode is a repeated, inherently sequential, comparatively lighter-weight-per-step operation (especially with KV caching).

Some advanced serving architectures separate these phases (sometimes called “disaggregated” prefill/decode serving) specifically to optimize each independently — allocating resources suited to each phase’s distinct cost profile, rather than treating the entire request as one undifferentiated computation.

21. Next Step

Next: Module 15 — Temperature, Top-K and Top-P — precisely how a token gets selected from the probability distribution (Module 5), and how each sampling parameter changes generation.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026