How an LLM Generates Text | TechByteByByte

Before you continue: three tools for this module

Token: a piece of text processed by the model.
Parameter: a learned number controlling the model’s transformations.
Inference: using the trained model without updating its parameters.

You do not need to memorize these yet. Use this map when the terms reappear.

Begin with the central question

What hidden problem does How an LLM Generates Text solve inside a real language-model system?

Keep that central question about How an LLM Generates Text in mind. The definitions, numbers, diagrams, and code examples below answer it one piece at a time.

prompt → tokens → forward pass → sampling → append token → repeat

1. What You Will Learn

Learning outcomes

Trace a prompt through tokenization, model computation, decoding, and repetition.
Explain why generation requires multiple forward passes.
Identify where sampling settings change output behavior.
Separate model generation from streaming, safety checks, and application code.

In one sentence

💡 Big picture

An LLM generates text one token at a time: predict, choose, append, and repeat until it reaches a stopping point.

2. Why This Module Exists

The problem this module solves

A finished answer can look as if it appeared all at once, but it was built through many model calls inside one generation loop.
Seeing the loop helps you understand streaming, latency, sampling, and stopping rules.

3. Intuition

an LLM never “writes a sentence” as one operation. It predicts one token, appends it to the growing sequence, and treats that longer sequence as brand-new input for the very next prediction — over and over, until it decides to stop (Module 15’s <EOS> mechanism) or hits a length limit.

4. Core Concept — The Complete Loop

Prompt
 ↓
Tokenization              (Module 2)
 ↓
Token IDs
 ↓
Embeddings                  (Module 4)
 ↓
Transformer                   (Module 4, 10)
 ↓
Logits                          (Module 5)
 ↓
Softmax
 ↓
Sampling / Selection               (Module 15)
 ↓
Next token
 ↓
Append token
 ↓
REPEAT — using the ENTIRE new sequence as input

Analogy: The Photocopy & Typing Loop Think of generating a story turn by turn as using an old-fashioned photocopier:

The Setup: You want to write a story. You type your prompt: “The sky is”.

Step 1: You feed this page into the photocopier. The machine runs a full scan (forward pass), prints a new page with the single word “is” appended at the bottom, and stops.

Step 2: To write the next word, you don’t just ask the machine for another word. You take that entire new page (“The sky is is”), feed it back into the tray, scan the whole thing from scratch, and it prints “sky”.

Every single word requires scanning the entire paper from page 1 again. This is why generation is sequential and scales with length.

📊 Visual Flowchart: The Autoregressive Text Generation Loop

Here is how new tokens append to the context matrix at successive generation steps:

graph TD
    classDef input fill:#3498db,stroke:#333,stroke-width:1px,color:#fff;
    classDef model fill:#e74c3c,stroke:#333,stroke-width:1px,color:#fff;
    classDef output fill:#2ecc71,stroke:#333,stroke-width:1px,color:#fff;

    subgraph Step1 ["Step 1: Predict first token"]
        In1["Prompt: ['the', 'sky', 'is']"]:::input --> Forward1["Forward Pass"]:::model
        Forward1 --> Prob1["Probabilities: ['is': 57%, 'blue': 7%]"]
        Prob1 --> Select1["Argmax Choice: 'is'"]:::output
    end

    subgraph Step2 ["Step 2: Predict second token"]
        Select1 --> Append1["Append word -> new input"]
        Append1 --> In2["Prompt: ['the', 'sky', 'is', 'is']"]:::input
        In2 --> Forward2["Forward Pass"]:::model
        Forward2 --> Prob2["Probabilities: ['is': 49%, 'sky': 38%]"]
        Prob2 --> Select2["Argmax Choice: 'is'"]:::output
    end

    subgraph Step3 ["Step 3: Predict third token"]
        Select2 --> Append2["Append word -> new input"]
        Append2 --> In3["Prompt: ['the', 'sky', 'is', 'is', 'is']"]:::input
        In3 --> Forward3["Forward Pass"]:::model
        Forward3 --> Prob3["Probabilities: ['sky': 38%, 'is': 25%]"]
        Prob3 --> Select3["Argmax Choice: 'sky'"]:::output
    end

5. How It Works — Step by Step

1. Tokenize the prompt (Module 2)
2. Run the FULL forward pass (Module 4, 5) -- producing a next-
   token probability distribution
3. SELECT a token from this distribution (Module 15 covers HOW)
4. APPEND this token to the sequence
5. Check STOPPING CONDITIONS: was an <EOS> token generated? Has
   a maximum length been reached?
6. If not stopping: go back to step 2, now with ONE MORE token
   of context than before -- run the ENTIRE forward pass again
7. Repeat until a stopping condition is met

Every single generated token requires a complete forward pass through the entire model — this repeated, full computation is directly why generation latency scales with the number of tokens generated (Module 14 covers the serving-side implications in full).

6. Mathematical Intuition

Read the mathematics as a story

prompt → tokens → forward pass → sampling → append token → repeat

First locate the input, operation, and output. Then treat the formula as a compact description of that journey rather than a collection of symbols to memorize.

Nothing new mathematically — this module’s value is entirely in observing Modules 5-6’s mechanism repeat, with the input sequence growing by exactly one token each iteration. The number of complete forward passes required to generate N tokens is exactly N.

7. Small Worked Example

Walk through the example

Name what each input represents.
Follow one transformation at a time.
Translate the result back into ordinary language.

The purpose is to reveal the mechanism, not merely display an answer.

Generating a 4-token response requires running the complete Transformer forward pass 4 separate times — once to predict each new token, each time with the previously-generated token now included as part of the input. This is why a longer requested response takes proportionally longer to generate.

8. Python Example

What the code will demonstrate

The code builds a tiny version of the mechanism, prints values you can inspect, and connects them to the worked example. Predict the direction of the result before running it.

Python symbols used below

NumPy (np) stores numeric vectors and matrices.
np.array(...) creates a numeric collection.
Library calls perform the same conceptual steps shown above at a larger scale.

# Build a small, inspectable example of How an LLM Generates Text.
# Follow the inputs, transformations, and output in order.
import numpy as np

def softmax(x):
    exp_x = np.exp(x - np.max(x))
    return exp_x / np.sum(exp_x)
def softmax_rows(x):
    exp_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
    return exp_x / np.sum(exp_x, axis=-1, keepdims=True)
def layer_norm(x, eps=1e-8):
    mean = x.mean(axis=-1, keepdims=True)
    std = x.std(axis=-1, keepdims=True)
    return (x - mean) / (std + eps)
def relu(x): return np.maximum(0, x)
def causal_mask(n): return np.triu(np.ones((n, n)), k=1).astype(bool)

np.random.seed(7)
d_model = 8
vocab = ["the", "sky", "is", "blue", "clear", "<EOS>"]
vocab_size = len(vocab)
embedding_table = np.round(np.random.randn(vocab_size, d_model) * 0.4, 3)
W_lm_head = np.round(np.random.randn(vocab_size, d_model) * 0.5, 2)

def positional_encoding(n, d):
    pos = np.arange(n)[:, np.newaxis]
    i = np.arange(d)[np.newaxis, :]
    angles = pos / np.power(10000, (2 * (i // 2)) / np.float32(d))
    pe = np.zeros((n, d))
    pe[:, 0::2] = np.sin(angles[:, 0::2])
    pe[:, 1::2] = np.cos(angles[:, 1::2])
    return pe

def decoder_block(x, seed):
    rng = np.random.RandomState(seed)
    n = x.shape[0]
    Wq, Wk, Wv, Wo = [rng.randn(d_model, d_model) * 0.3 for _ in range(4)]
    Q, K, V = x @ Wq, x @ Wk, x @ Wv
    scores = Q @ K.T / np.sqrt(d_model)
    scores[causal_mask(n)] = -np.inf
    attn_out = (softmax_rows(scores) @ V) @ Wo
    x = layer_norm(x + attn_out)
    W1 = rng.randn(d_model, d_model * 2) * 0.3
    W2 = rng.randn(d_model * 2, d_model) * 0.3
    ffn_out = relu(x @ W1) @ W2
    return layer_norm(x + ffn_out)

def forward_pass(token_ids):
    x = embedding_table[token_ids] + positional_encoding(len(token_ids), d_model)
    for i in range(2):
        x = decoder_block(x, seed=i)
    logits = W_lm_head @ x[-1]
    return softmax(logits)

# --- Full iterative generation loop ---
prompt = ["the", "sky", "is"]
generated = list(prompt)
max_new_tokens = 4

print(f"Prompt: {prompt}\n")
for step in range(max_new_tokens):
    token_ids = [vocab.index(w) for w in generated]
    probs = forward_pass(token_ids)
    next_id = np.argmax(probs)
    next_token = vocab[next_id]
    print(f"Step {step+1}: input={generated} -> predicted: '{next_token}' (p={probs[next_id]:.4f})")
    generated.append(next_token)
    if next_token == "<EOS>":
        print("  <EOS> generated -- STOPPING")
        break

print(f"\nFinal generated sequence: {generated}")
print(f"Number of FORWARD PASSES required: {step+1}")

Expected Output:

Prompt: ['the', 'sky', 'is']

Step 1: input=['the', 'sky', 'is'] -> predicted: 'is' (p=0.5795)
Step 2: input=['the', 'sky', 'is', 'is'] -> predicted: 'is' (p=0.4978)
Step 3: input=['the', 'sky', 'is', 'is', 'is'] -> predicted: 'sky' (p=0.3881)
Step 4: input=['the', 'sky', 'is', 'is', 'is', 'sky'] -> predicted: 'is' (p=0.3780)

Final generated sequence: ['the', 'sky', 'is', 'is', 'is', 'sky', 'is']
Number of FORWARD PASSES required: 4

9. How It Works

Exactly 4 complete forward passes were run to generate 4 tokens — the input sequence genuinely grows by one token each step (3 → 4 → 5 → 6 tokens long), and the entire growing sequence is reprocessed from scratch at every single step (Module 14’s KV cache exists specifically to avoid the redundant recomputation this implies).
This particular (deliberately untrained, random-weight) model fell into a repetition loop — repeatedly predicting “is.” This is an honest, genuine illustration of a real, well-documented failure mode: greedy decoding (always picking the highest-probability token) can get stuck in repetitive loops, especially with weaker models or certain prompts. Module 15 covers sampling strategies (temperature, top-k, top-p) that specifically help avoid this.

10. How Is This Used in Modern AI?

Trace it through a real model call

user message → assembled context → LLM computation → decoded output → application checks

This topic affects one stage of that path; it is not the complete product. Hosted GPT- and Gemini-style applications also add instructions, safety systems, retrieval, tools, serving infrastructure, and evaluation around the model.

🤖 How Is This Used in Modern AI?

This exact loop, run with a real trained model, is what generates every response from every LLM product. The “typing” effect you see in chat interfaces isn’t cosmetic — it’s a direct, literal reflection of tokens being generated one full forward pass at a time and streamed to the display as each becomes available.

11. How Is This Used in Agentic AI?

Separate the model from the runtime

goal + state + tool results → LLM proposal → runtime validation → execution or response

The LLM proposes text or a structured action. Ordinary application code controls permissions, tools, retries, memory, and execution.

Direct relevance to Agentic AI: Very High. Every piece of an agent’s output — reasoning text, tool call arguments, final responses — is produced by this exact repeated loop.

Understanding that generation is inherently iterative and sequential (each token depends on the previous ones) directly explains why longer agent outputs take proportionally longer and why streaming responses to a user, token by token, is a natural fit for this generation process.

12. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: assuming a full response is generated in one computation.

Why it is incorrect: As demonstrated directly, it’s N complete forward passes for N generated tokens — never a single, one-shot operation.

⚠️ Mistake

Incorrect idea: assuming repetition loops indicate something is “broken” architecturally.

Why it is incorrect: As shown directly, this is a genuine, known consequence of greedy decoding specifically — not a sign the underlying mechanism has failed; it’s addressed by sampling strategy choices (Module 15), not architectural changes.

⚠️ Mistake

Incorrect idea: believing the model has access to some “plan” for the whole response before starting.

Why it is incorrect: Nothing in this mechanism produces or uses a pre-formed plan — each token is decided based solely on everything generated (or provided) so far, one step at a time.

13. Important Distinctions

One Forward Pass (Module 5)	The Full Generation Loop (this module)
Predicts ONE next-token distribution	Repeats the forward pass once per generated token
A single computation	An iterative process, `N` passes for `N` tokens

Greedy Decoding	Sampling-Based Decoding (Module 15)
Always picks the highest-probability token	Introduces controlled randomness — can avoid repetition loops

14. When to Use

This iterative loop is the universal mechanism — not a design choice with alternatives. What IS a design choice is the selection strategy at each step (Module 15) and whether/how to optimize the repeated computation (Module 14’s KV cache, Module 24’s broader optimization techniques).

15. When Not to Use

Not applicable — every autoregressive LLM generates text this way.

16. Production Considerations

Generation latency scales directly with output length — verified directly: 4 tokens required 4 full forward passes; a 500-token response requires 500.
Repetition loops are a real, monitorable production issue — production systems often include repetition penalties or specific sampling strategies (Module 15) precisely to mitigate this genuine failure mode.
Streaming output to users (showing tokens as they’re generated, rather than waiting for the full response) is a natural fit for this loop’s structure and a standard practice for improving perceived latency.

17. What You Should Remember

Generation is fundamentally iterative: one complete forward pass per generated token, verified directly with a real 4-step trace.
The entire sequence is reprocessed at each step in this naive version — directly motivating Module 14’s KV cache optimization.
Repetition loops are a real, documented failure mode of greedy decoding — not a sign of a broken mechanism, and directly addressed by Module 15’s sampling strategies.

18. Interview Questions

Beginner

Q: Why does generating a longer response from an LLM take proportionally longer than generating a shorter one?

Ans: Generation is iterative — each new token requires one complete forward pass through the entire model, using the growing sequence (prompt plus everything generated so far) as input. Verified directly: generating 4 tokens required exactly 4 full forward passes.

A longer response simply requires more of these repeated passes.

Intermediate

Q: What causes an LLM to sometimes get stuck repeating the same word or phrase during generation?

Ans: This is a genuine, documented failure mode of greedy decoding (always selecting the single highest-probability token at each step) — demonstrated directly in this module, where an untrained model repeatedly predicted the same token, creating a repetition loop.

Sampling-based decoding strategies (Module 15) that introduce controlled randomness are a standard mitigation, since they don’t deterministically re-select the exact same highest-probability token every time.

Advanced

Q: Explain precisely why the model has no access to a “plan” for its entire response before it starts generating.

Ans: As traced directly in this module’s loop, each token’s prediction depends solely on the current sequence — the prompt plus whatever tokens have been generated so far in this specific run.

There’s no separate planning step, no pre-computed outline, and no mechanism storing intended future content anywhere in this process. Any apparent “planning” in a response’s structure emerges entirely from the model’s learned tendency (from training, Module 8) to generate token sequences that are locally coherent given everything so far — not from an explicit upfront plan.

Scenario

Q: A production chatbot occasionally produces responses that repeat the same sentence over and over. Using this module, what would you investigate?

Ans: I’d first check the decoding/sampling configuration — this is a classic symptom of greedy or near-greedy decoding, demonstrated directly in this module.

I’d check whether temperature, top-k, or top-p (Module 15) are configured to introduce enough controlled randomness to avoid the model deterministically re-selecting the same high-probability continuation repeatedly, and consider whether a repetition penalty is applied. This is a decoding-strategy issue, not evidence of a fundamentally broken model.

AI Engineering

Q: Why does understanding that generation is a repeated, full forward-pass loop matter for reasoning about LLM API latency and cost?

Ans: Because it directly explains why response time scales with output length (more tokens requested means more repeated forward passes, verified directly), and why the naive version of this loop — reprocessing the entire growing sequence from scratch every step — would be extremely wasteful at scale.

This is precisely the motivation for Module 14’s KV cache: caching the results of already-processed tokens so each new step only needs to process the newest token, rather than redoing the full sequence’s computation every single time.

19. Next Step

Next: Module 8 — Pretraining — how the model’s weights, which this entire mechanism depends on, actually get learned in the first place, connecting directly to your Neural Networks and Optimization courses.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026