Decoder-Only LLMs | TechByteByByte

Before you continue: three tools for this module

Token: a piece of text processed by the model.
Parameter: a learned number controlling the model’s transformations.
Inference: using the trained model without updating its parameters.

You do not need to memorize these yet. Use this map when the terms reappear.

Begin with the central question

What hidden problem does Decoder-Only LLMs solve inside a real language-model system?

Keep that central question about Decoder-Only LLMs in mind. The definitions, numbers, diagrams, and code examples below answer it one piece at a time.

visible earlier tokens → causal Transformer → next-token distribution

1. What You Will Learn

Learning outcomes

Define a decoder-only Transformer and its causal visibility rule.
Explain why future tokens must be masked during both training and generation.
Compare decoder-only, encoder-only, and encoder-decoder designs.
Trace how one architecture supports prompting and open-ended generation.

In one sentence

💡 Big picture

A decoder-only LLM reads only the tokens already available and uses them to predict the next token.

2. Why This Module Exists

The problem this module solves

During generation, future tokens do not exist yet.
Causal masking teaches the model to follow the same rule during training instead of secretly looking ahead.

3. Intuition

you already know encoder vs. decoder architecture from the Transformers course. The specific LLM-relevant fact: at generation time, a token being predicted genuinely doesn’t exist yet — so the model must never be allowed to “peek” at future tokens, even during training when the full sequence is technically available. Causal masking enforces this rule directly, mechanically.

Analogy: The Single-Direction Mirror Maze & The Sealed Envelopes Think of causal masking in terms of keeping students honest during an exam:

The Setup: You place 4 desks in a straight line: Alice (0), Bob (1), Charlie (2), and David (3).

The Peeking Rule (Bidirectional Attention): Without partitions, everyone can turn around and look at everyone else’s test sheets. This is how encoders work — great for understanding a complete, fixed sentence.

The Mirror Rule (Causal Attention): You install single-direction mirrors between the desks.

Alice (0) can only see her own desk.

Bob (1) can see Alice’s sheet and his own, but Charlie and David are blocked behind opaque walls.

David (3) sits at the end and can see everyone’s sheets.

The future answers are sealed in envelopes behind each desk. Since future tokens genuinely don’t exist yet at generation time (inference), causal masking forces the model to learn representations without ever peeking forward during training.

📊 Visual Chart: The Causal Attention Mask Matrix

Here is how query indices can attend to key indices (False = Allowed, True = Masked/Blocked):

flowchart TB
    %% Causal attention grows from left to right: each token can see itself and earlier tokens only.
    T0["Token 0: I<br/>can see: I"]
    T1["Token 1: love<br/>can see: I, love"]
    T2["Token 2: machine<br/>can see: I, love, machine"]
    T3["Token 3: learning<br/>can see: I, love, machine, learning"]

    T0 --> T1 --> T2 --> T3

(Below is the standard mathematical representation of causal self-attention queries and keys)

graph TD
    classDef allowed fill:#2ecc71,stroke:#333,stroke-width:1px,color:#fff;
    classDef blocked fill:#e74c3c,stroke:#333,stroke-width:1px,color:#fff;

    subgraph AttentionMap ["Query (rows) vs. Key (columns) Attention Matrix"]
        Row0_Col0["'I' to 'I'<br>(Allowed)"]:::allowed
        Row0_Col1["'I' to 'love'<br>(MASKED)"]:::blocked
        Row0_Col2["'I' to 'machine'<br>(MASKED)"]:::blocked
        Row0_Col3["'I' to 'learning'<br>(MASKED)"]:::blocked

        Row1_Col0["'love' to 'I'<br>(Allowed)"]:::allowed
        Row1_Col1["'love' to 'love'<br>(Allowed)"]:::allowed
        Row1_Col2["'love' to 'machine'<br>(MASKED)"]:::blocked
        Row1_Col3["'love' to 'learning'<br>(MASKED)"]:::blocked

        Row2_Col0["'machine' to 'I'<br>(Allowed)"]:::allowed
        Row2_Col1["'machine' to 'love'<br>(Allowed)"]:::allowed
        Row2_Col2["'machine' to 'machine'<br>(Allowed)"]:::allowed
        Row2_Col3["'machine' to 'learning'<br>(MASKED)"]:::blocked

        Row3_Col0["'learning' to 'I'<br>(Allowed)"]:::allowed
        Row3_Col1["'learning' to 'love'<br>(Allowed)"]:::allowed
        Row3_Col2["'learning' to 'machine'<br>(Allowed)"]:::allowed
        Row3_Col3["'learning' to 'learning'<br>(Allowed)"]:::allowed
    end

4. Core Concept — Recap and Focus

Encoder:      bidirectional attention -- every position sees
              EVERY other position, including LATER ones
              (Transformers course)

Decoder:        CAUSAL attention -- every position sees only
              ITSELF and EARLIER positions, never later ones
              (Transformers course)

Decoder-only:     GPT-style LLMs use ONLY the decoder side, with
              causal attention throughout -- NO separate encoder,
              NO cross-attention (Transformers course)

Why modern LLMs use decoder-only specifically (already established in the Transformers course, restated for this course’s context): the core LLM task — predict the next token given everything so far — has no separate “source” sequence requiring bidirectional encoding. Prompt and generated response form one continuous, causally-processed sequence.

5. How It Works — Step by Step

1. For a sequence of N tokens, causal attention computes an N x N
   attention pattern, but MASKS OUT (sets to -infinity BEFORE
   softmax, Transformers course) any position (i,j) where j > i
   -- i.e., where the KEY position comes AFTER the QUERY position
2. After softmax, these masked positions receive EXACTLY ZERO
   attention weight
3. This means: position i's representation can ONLY be built from
   information at positions 0 through i -- NEVER from position
   i+1 onward
4. This restriction applies BOTH during training (Module 8, where
   the whole sequence is technically available but must still be
   masked) AND during inference (Module 7, where future tokens
   literally don't exist yet)

6. Mathematical Intuition

Read the mathematics as a story

visible earlier tokens → causal Transformer → next-token distribution

First locate the input, operation, and output. Then treat the formula as a compact description of that journey rather than a collection of symbols to memorize.

Nothing new beyond the Transformers course’s causal masking mechanism — this module’s value is in confirming, with a fresh concrete example, that the masking pattern behaves exactly as expected: strictly triangular, each position seeing itself and only earlier positions.

7. Small Worked Example

Walk through the example

Name what each input represents.
Follow one transformation at a time.
Translate the result back into ordinary language.

The purpose is to reveal the mechanism, not merely display an answer.

For “I love machine learning,” the word “love” (position 1) should be able to attend to itself and “I” (position 0), but never to “machine” or “learning” (positions 2-3) — even though, during training, those later words are literally present in the same input sequence. Causal masking is what enforces this restriction mechanically.

8. Python Example

What the code will demonstrate

The code builds a tiny version of the mechanism, prints values you can inspect, and connects them to the worked example. Predict the direction of the result before running it.

Python symbols used below

NumPy (np) stores numeric vectors and matrices.
np.array(...) creates a numeric collection.
Library calls perform the same conceptual steps shown above at a larger scale.

# Build a small, inspectable example of Decoder-Only LLMs.
# Follow the inputs, transformations, and output in order.
import numpy as np

sentence = ["I", "love", "machine", "learning"]
n = len(sentence)

causal_mask = np.triu(np.ones((n, n)), k=1).astype(bool)

print("Sentence:", sentence)
print("\nWhat each position is ALLOWED to attend to (causal mask):\n")
for i, word in enumerate(sentence):
    allowed = [sentence[j] for j in range(n) if not causal_mask[i, j]]
    print(f"  Position {i} ('{word}'): can attend to {allowed}")

print("\nCausal mask matrix (True = MASKED/blocked, False = allowed):")
print(causal_mask)

print("\n--- TRAINING: entire sequence processed AT ONCE (parallel) ---")
print("All 4 positions' predictions computed in ONE forward pass,")
print("each still respecting the causal mask (no position sees the future).")

print("\n--- INFERENCE: tokens generated ONE AT A TIME (sequential) ---")
for i in range(1, n+1):
    print(f"  Step {i}: sees {sentence[:i]} -> predicts next token")

Expected Output:

Sentence: ['I', 'love', 'machine', 'learning']

What each position is ALLOWED to attend to (causal mask):

  Position 0 ('I'): can attend to ['I']
  Position 1 ('love'): can attend to ['I', 'love']
  Position 2 ('machine'): can attend to ['I', 'love', 'machine']
  Position 3 ('learning'): can attend to ['I', 'love', 'machine', 'learning']

Causal mask matrix (True = MASKED/blocked, False = allowed):
[[False  True  True  True]
 [False False  True  True]
 [False False False  True]
 [False False False False]]

--- TRAINING: entire sequence processed AT ONCE (parallel) ---
All 4 positions' predictions computed in ONE forward pass,
each still respecting the causal mask (no position sees the future).

--- INFERENCE: tokens generated ONE AT A TIME (sequential) ---
  Step 1: sees ['I'] -> predicts next token
  Step 2: sees ['I', 'love'] -> predicts next token
  Step 3: sees ['I', 'love', 'machine'] -> predicts next token
  Step 4: sees ['I', 'love', 'machine', 'learning'] -> predicts next token

9. How It Works

The allowed-attention list grows by exactly one word per position — position 3 (“learning”) can see all four words, while position 0 (“I”) can see only itself — a strictly triangular pattern, verified directly.
The mask matrix is False (allowed) exactly on and below the diagonal, True (blocked) exactly above it — the precise, mechanical definition of “no position sees the future.”
Training vs. inference use the identical causal restriction, just computed differently for efficiency: training processes the whole sequence in one parallel pass (Transformers course, and your Neural Networks/Optimization courses’ batch training), with the mask ensuring no position illegitimately uses future information even though it’s technically present in the batch. Inference genuinely doesn’t have future tokens available at all — they haven’t been generated yet — so the mask’s restriction is automatically satisfied by the sequential nature of generation itself (Module 7).

10. How Is This Used in Modern AI?

Trace it through a real model call

user message → assembled context → LLM computation → decoded output → application checks

This topic affects one stage of that path; it is not the complete product. Hosted GPT- and Gemini-style applications also add instructions, safety systems, retrieval, tools, serving infrastructure, and evaluation around the model.

🤖 How Is This Used in Modern AI?

Every GPT-style LLM — including every general-purpose model you’ll ever call via API — uses exactly this causal masking pattern throughout every attention computation, at every layer. This is not an occasional feature; it’s the structural foundation making autoregressive generation (Module 7) valid and correct rather than “cheating” by seeing the future during training.

11. How Is This Used in Agentic AI?

Separate the model from the runtime

goal + state + tool results → LLM proposal → runtime validation → execution or response

The LLM proposes text or a structured action. Ordinary application code controls permissions, tools, retries, memory, and execution.

Direct relevance to Agentic AI: High, foundational. An agent’s LLM processes its entire assembled context (system prompt, history, tool results, new instruction) as one long causally-masked sequence — every position can only “see” what came before it in that specific assembled order.

This is precisely why the order in which context is assembled for an agent genuinely matters (a detail also covered in the Transformers course).

12. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: assuming causal masking only matters during inference.

Why it is incorrect: As emphasized directly, it’s equally essential during training — without it, a model training on the full sequence in parallel could trivially “cheat” by looking at the very token it’s supposed to predict.

⚠️ Mistake

Incorrect idea: believing decoder-only means “no encoder exists anywhere in the pipeline.”

Why it is incorrect: As covered in the Transformers course: there’s no separate encoder component — but the model still builds understanding of the prompt using the same causal self-attention mechanism applied uniformly to the whole sequence.

⚠️ Mistake

Incorrect idea: thinking causal masking is unique to LLMs specifically, distinct from what the Transformers course covered.

Why it is incorrect: It’s the exact same mechanism — this module verifies it, doesn’t introduce a new variant.

13. Important Distinctions

Encoder (Bidirectional)	Decoder (Causal)
Every position sees every other position	Every position sees only itself and earlier positions — verified directly
Suited to understanding a fixed, complete input	Suited to autoregressive generation

Training (this module’s causal masking)	Inference (this module’s causal masking)
Whole sequence processed in parallel, masked	Tokens genuinely don’t exist yet — sequential by necessity
Same restriction, enforced explicitly via masking	Same restriction, naturally satisfied by generation order

14. When to Use

Decoder-only, causally-masked architecture is the standard, near- universal choice for general-purpose autoregressive LLMs (Module 1, and the Transformers course’s dedicated module on this exact topic).

15. When Not to Use

Full bidirectional (encoder-style) attention remains appropriate for tasks requiring understanding of a complete, already-available input with no generation component — classification or extraction tasks (NLP course) rather than open-ended text generation.

16. Production Considerations

Causal masking has real computational implications — the KV cache (Module 14, Transformers course) specifically exploits the fact that earlier positions’ Key/Value computations never need to change once computed, precisely because causal masking guarantees they never depend on later tokens.
Context assembly order matters directly — since causal attention can only look backward, placing critical information late in an assembled prompt (after less important content) can affect how effectively the model can integrate it into the final response.

17. What You Should Remember

Causal attention ensures every position sees only itself and earlier positions — verified directly with a real, complete masking pattern for a 4-word sentence.
This restriction applies identically during training and inference — training enforces it explicitly via masking; inference satisfies it naturally, since future tokens don’t exist yet.
Decoder-only architecture — no separate encoder, no cross- attention, purely causal self-attention — is the standard choice for general-purpose LLMs, precisely because next-token prediction has no separate “source” sequence requiring bidirectional encoding.

18. Interview Questions

Beginner

Q: Why do decoder-only LLMs use causal attention instead of the full, bidirectional attention an encoder uses?

Ans: The model’s core task is predicting the next token given everything before it — allowing a position to see tokens that come after it would mean the model could “cheat” during training by directly seeing the answer it’s supposed to predict, and during actual generation, later tokens genuinely don’t exist yet since they haven’t been produced.

Causal masking mechanically prevents any position from attending to later positions, ensuring training and inference behave consistently.

Intermediate

Q: Why is causal masking necessary during training, when the entire sequence is technically available in the batch?

Ans: Even though the full sequence exists in the training batch (for parallel, efficient processing), the model must learn to predict each position using only the information that would genuinely be available at that point during real generation — i.e., only earlier positions.

Without causal masking, a position during training could trivially attend to the very token it’s being trained to predict, making the training signal meaningless and producing a model that couldn’t actually generate coherently at inference time, when future tokens truly aren’t available.

Advanced

Q: Explain, using the verified example from this module, why causal masking produces a strictly triangular attention pattern.

Ans: Causal masking blocks any attention connection where the key position comes after the query position (j > i).

Verified directly: position 0 could only attend to itself (a single allowed position), position 1 could attend to positions 0-1, position 2 to positions 0-2, and position 3 (the last) could attend to all four positions.

This strictly increasing pattern of allowed attention — each position allowed exactly one more connection than the previous — is precisely what produces the lower-triangular structure in the mask matrix (False on and below the diagonal, True above it).

Scenario

**Q: A team is deciding whether to use a decoder-only or encoder-decoder architecture for a new document classification system, where the entire document is always available upfront.

What would you recommend, using this module’s concepts?** A: Since the full document is always available upfront (not being generated token by token) and the task is classification (not generation), an encoder-style architecture with full, bidirectional attention would be a more natural architectural fit — every position could benefit from seeing the entire document, including text that comes later, which causal masking would otherwise prevent.

A decoder-only model could still be adapted for this task, but its causal restriction isn’t providing any benefit here, since there’s no generation process requiring it.

AI Engineering

Q: How does causal masking directly enable the KV cache optimization covered in the Transformers course? A: Because causal masking guarantees a position’s Key and Value vectors never depend on any LATER token (only on itself and earlier positions), once a token’s Key/Value has been computed, it will never need to change as generation continues — this is precisely what makes it safe to cache and reuse those values across subsequent generation steps (Module 14 of this course, and the Transformers course’s KV cache module), rather than recomputing them from scratch at every single step.

19. Next Step

Next: Module 12 — Model Parameters and Architecture — what parameter count actually means, and what a “70B parameter model” statement actually tells you.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026