TechByteByByte

Encoder vs Decoder vs Decoder-Only LLMs

A deeper look at encoder, decoder, and encoder-decoder architectural tradeoffs, with a verified cross-attention computation, and precisely why modern LLMs overwhelmingly use decoder-only architectures.

#Transformers#Encoder#Decoder#Cross-Attention#LLM Architecture#AI

Begin with the central question

Why do BERT-style understanding models and GPT-style generation models use different attention directions?

Essential words

An encoder can use bidirectional context over available input. A decoder-only model uses causal self-attention. Cross-attention lets a decoder read a separate encoder output.

What You Will Understand

A deeper architectural comparison of encoder, decoder, and encoder-decoder Transformers — including a real, verified cross-attention computation, the mechanism unique to encoder-decoder models — and precisely why modern general-purpose LLMs overwhelmingly settled on decoder-only, with honest treatment of the tradeoffs rather than a claim that one architecture is universally best.

encoder: full context | decoder: past-only | encoder-decoder: input plus generated past

The problem this module solves

Module 12 introduced the three architecture families at a survey level. This module exists to go deeper on the one mechanism that’s genuinely new here — cross-attention — and to answer the practical question every AI engineer eventually asks: why does virtually every LLM you actually use turn out to be decoder-only?

Build the intuition

an encoder builds understanding by looking at everything at once, like reading an entire reference document before answering a question about it. A decoder builds output progressively, one word at a time, only ever looking backward at what’s already been written — like writing an essay where you can revise your understanding of what to write next, but never peek ahead at your own unwritten conclusion.

4. Real-World Analogy

Think of a human translator working live: they first read (or listen to) the entire source sentence — full context, both directions — building a complete understanding (the encoder’s job).

Then they produce the translated sentence word by word, each new word informed both by their full understanding of the source (cross-attention to the encoder) and by what they’ve already said in the translation so far (causal self- attention within the decoder).

Analogy: The Simultaneous Live Translator (Cross-Attention Linking) Think of translation systems in terms of an international conference booth:

  • The Encoder (French Listener): Listens to the entire incoming French sentence “le chat sit sur le paillasson” simultaneously. They write down a contextual outline sheet containing the subject, verb, and object (The Encoder Output keys and values).
  • The Decoder (English Speaker): Generates the English translation token-by-token:
    • First, they output “The”.
    • Now, to output the next word, they run two checks:
      • Causal Self-Attention: What did I just say? I said “The”.
      • Cross-Attention: They point their Query (“The”) to the listener’s French outline sheet (Keys/Values). They match “The” to French “le” and retrieve the neighboring subject “chat” (highest weight) to output “cat”.

📊 Visual Chart: Encoder-Decoder Cross-Attention Sequence

Here is the interaction flow between the bidirectional encoder stack and the causal decoder:

sequenceDiagram
    participant Encoder as Encoder Stack (Source: 'le chat')
    participant Cross as Cross-Attention Layer
    participant Decoder as Decoder Stack (Target: 'the')




    Note over Encoder: Bidirectional Self-Attention<br>(Reads both words simultaneously)
    Encoder->>Cross: Sends Key (K_src) & Value (V_src) Matrices




    Note over Decoder: Causal Self-Attention<br>(Reads generated history)
    Decoder->>Cross: Sends Query (Q_tgt) representing current token 'the'




    Note over Cross: Computes softmax( Q_tgt @ K_src.T / sqrt(d_k) ) V_src
    Cross->>Decoder: Returns Contextualized Translation Vector

5. Core Concept

Encoder:      bidirectional self-attention
              (every position sees every other position)




Decoder:      causal self-attention (Module 6)
              (every position sees only itself + earlier positions)
              PLUS, in encoder-decoder models specifically:
              cross-attention to the encoder's output




Decoder-only: causal self-attention ONLY -- no separate encoder,
              no cross-attention at all

Cross-attention, precisely

Self-attention:    Query, Key, AND Value all come from the SAME
                    sequence (Module 3-5)




Cross-attention:    Query comes from the DECODER (what it's
                    currently trying to generate); Key and Value
                    come from the ENCODER's output (the full,
                    already-processed source sequence)

6. How It Works — Step by Step

Encoder-decoder generation, one step:

1. The ENCODER processes the complete source sequence ONCE,
   using full (non-causal) self-attention -- producing a
   contextual representation of the entire source
2. The DECODER begins generating the target sequence
3. At each decoder step, TWO attention computations happen:
   a. CAUSAL SELF-ATTENTION over the target tokens generated
      SO FAR (Module 6 -- can't see future target tokens)
   b. CROSS-ATTENTION: the decoder's current Query attends to
      the ENCODER's Key and Value -- pulling relevant
      information from the FULL source sequence
4. Both attention outputs feed into the rest of the decoder
   block (FFN, residuals, norm -- Module 9)
5. Repeat for each new target token, causally

Decoder-only generation simply skips steps 1 and 3b entirely — there is no separate source sequence to encode, and no cross-attention. Only step 3a (causal self-attention over everything generated/provided so far) happens, applied uniformly to the entire sequence.

7. Mathematical Intuition

Read the mathematics as a story

The attention formula stays the same; visibility and the sources of Q/K/V change. Encoders read available input bidirectionally, causal decoders read only the generated past, and cross-attention connects two sequences.

encoder self-attention: Q,K,V from source
decoder self-attention: Q,K,V from generated prefix
cross-attention: Q from decoder; K,V from encoder

Cross-attention uses the exact same formula as self-attention (Module 5) — softmax(QK^T / √d_k)V — the only difference is where Q, K, V come from. In self-attention, all three are projections of the same input. In cross-attention, Q is a projection of the decoder’s current state, while K and V are projections of the encoder’s output — different sequences entirely, computed once for the encoder and reused across every decoder step.

8. Small Worked Example

Walk through the example

  1. Encode a short source sentence. 2. Create a decoder Query. 3. Compare it with source Keys. 4. Mix source Values to guide the next output.

Translating French “le chat” to English “the cat”: the encoder processes “le chat” once, producing a representation of both words. When the decoder has generated “the” and needs to decide the next word, its cross-attention Query (from “the”) is compared against Keys derived from both “le” and “chat” — likely attending most strongly to “chat,” since that’s the source word most relevant to generating “cat” next.

9. Python / NumPy Example

What the code will demonstrate

This small NumPy example makes Encoder vs Decoder vs Decoder-Only LLMs visible with inspectable numbers and shapes. Read it in three passes: identify each input, follow the transformation line by line, and connect the printed output to the diagram above. The arrays are intentionally tiny teaching values; unless the text explicitly says otherwise, they are not weights or measurements from GPT, Gemini, or another trained model.

# The arrays are intentionally small so each transformation can be inspected.
# Printed values illustrate the mechanism; they are not trained-model measurements.
import numpy as np




def softmax(x, axis=-1):
    exp_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return exp_x / np.sum(exp_x, axis=axis, keepdims=True)




np.random.seed(9)
d_model = 4




# Encoder processes the SOURCE sequence ("le chat")
source_tokens = ["le", "chat"]
encoder_output = np.round(np.random.randn(len(source_tokens), d_model) * 0.5, 3)
print("Encoder output (representation of SOURCE sequence):\n", encoder_output)




# Decoder has generated "the" so far, deciding the NEXT token
target_so_far = ["the"]
decoder_hidden = np.round(np.random.randn(len(target_so_far), d_model) * 0.5, 3)
print("\nDecoder's own hidden state (target so far):\n", decoder_hidden)




# --- Self-attention: Q, K, V all from the decoder ---
Wq_self = np.round(np.random.randn(d_model, d_model) * 0.4, 2)
Wk_self = np.round(np.random.randn(d_model, d_model) * 0.4, 2)
Wv_self = np.round(np.random.randn(d_model, d_model) * 0.4, 2)




Q_self = decoder_hidden @ Wq_self
K_self = decoder_hidden @ Wk_self
V_self = decoder_hidden @ Wv_self
self_scores = Q_self @ K_self.T / np.sqrt(d_model)
self_attn_out = softmax(self_scores, axis=-1) @ V_self
print("\nDecoder SELF-attention output:\n", np.round(self_attn_out, 4))




# --- Cross-attention: Q from decoder, K/V from ENCODER ---
Wq_cross = np.round(np.random.randn(d_model, d_model) * 0.4, 2)
Wk_cross = np.round(np.random.randn(d_model, d_model) * 0.4, 2)
Wv_cross = np.round(np.random.randn(d_model, d_model) * 0.4, 2)




Q_cross = decoder_hidden @ Wq_cross      # Query FROM THE DECODER
K_cross = encoder_output @ Wk_cross       # Key FROM THE ENCODER
V_cross = encoder_output @ Wv_cross       # Value FROM THE ENCODER




cross_scores = Q_cross @ K_cross.T / np.sqrt(d_model)
cross_weights = softmax(cross_scores, axis=-1)
cross_attn_out = cross_weights @ V_cross
print("\nDecoder CROSS-attention output:\n", np.round(cross_attn_out, 4))
print("Cross-attention weights (decoder's 'the' attending to each SOURCE token):")
for i, tok in enumerate(source_tokens):
    print(f"  '{tok}': {cross_weights[0, i]:.4f}")

Expected Output:

Encoder output (representation of SOURCE sequence):
 [[ 0.001 -0.145 -0.558 -0.006]
 [-0.189 -0.241 -0.759 -0.245]]




Decoder's own hidden state (target so far):
 [[-0.12  -0.324  0.318  0.87 ]]




Decoder SELF-attention output:
 [[-0.213  -0.5399 -0.0043 -0.2476]]




Decoder CROSS-attention output:
 [[-0.2321 -0.0052 -0.1469 -0.1024]]
Cross-attention weights (decoder's 'the' attending to each SOURCE token):
  'le': 0.4954
  'chat': 0.5046

10. How It Works

  • The self-attention computation uses only the decoder’s own hidden state (decoder_hidden) for Q, K, and V — exactly Module 3-5’s mechanism, just applied within the decoder.
  • The cross-attention computation uses the decoder’s hidden state only for Q, while K and V come from the encoder’s output — structurally different data sources feeding the same attention formula.
  • The cross-attention weights (~0.495 for “le”, ~0.505 for “chat”) show the decoder’s current position drawing information from both source tokens, roughly evenly in this untrained example — in a genuinely trained translation model, you’d expect this to sharpen toward whichever source word is most directly relevant to the next target word being generated.

11. How Is This Used in Modern AI?

Where this concept lives

Follow the concept at three levels: inside the model, where the computation happens; inside the AI product, where that computation supports a visible feature; and inside production, where engineers measure speed, memory, quality, and failure cases. The details below connect those levels.

🤖 How Is This Used in Modern AI?

Decoder-only became the dominant architecture for general-purpose LLMs — but it’s worth being precise about why, rather than treating it as an unqualified “decoder-only wins.”

Why decoder-only became dominant for general-purpose LLMs:

  • A general-purpose LLM’s core task — predicting the next token given everything so far — has no inherent “distinct source sequence” requiring separate encoding. Prompt and generated response can be treated as one continuous, causally-processed sequence.
  • This avoids the added architectural complexity of a separate encoder and cross-attention entirely.
  • Decoder-only models trained at scale on next-token prediction have proven to generalize remarkably well to a huge range of tasks (question-answering, summarization, translation, reasoning) — tasks that historically might have used encoder-only or encoder-decoder architectures — via prompting alone, without needing task-specific architectural changes.

This is not a claim that decoder-only is universally superior:

  • For tasks with a genuinely fixed, complete input requiring pure understanding (not generation), encoder-only models remain a reasonable, often more efficient choice.
  • For tasks with a very clear, structurally distinct source-and-target relationship, encoder-decoder architectures still have genuine advantages in some settings.

12. How Is This Used in LLMs?

Trace one model call

User text → tokens → Transformer computation → output-token probabilities
          this topic affects one part of that computation

An LLM does not apply this idea as a separate magic step. It uses it as part of the repeated numerical pipeline that transforms token vectors and produces the next-token probabilities.

Every general-purpose LLM you interact with via API — GPT-style models, Claude — is decoder-only. There is no separate encoder processing your prompt with bidirectional attention; your entire prompt and the model’s generated response are processed as one continuous sequence, with causal self-attention throughout (Module 6), exactly as demonstrated in Module 12’s stacked-block example.

Real systems you can recognize

Hugging Face documents BERT as an encoder model and provides causal language-model classes for generation. GPT-style assistants are decoder-only; translation models such as T5 use encoder-decoder structure.

13. How Is This Used in Agentic AI?

Trace one agent step

Goal + history + tool results
              ↓
     LLM processes the context
              ↓
Suggested answer or tool call
              ↓
Agent runtime validates and executes it

This distinction matters: the Transformer helps produce the proposal, while the surrounding agent software controls tools, permissions, retries, memory, and execution.

Direct relevance to Agentic AI: High. An agent’s entire prompt — system instructions, conversation history, retrieved documents, tool definitions and results — is fed into a decoder-only LLM as one unified, causally-processed sequence, not through a separate encoder.

This is part of why prompt ordering matters so much in agent system design (Module 8’s positional information directly): everything the model “sees” is processed causally, in the order it’s presented.

When this knowledge is useful

Use Encoder vs Decoder vs Decoder-Only LLMs when you need to explain, implement, debug, evaluate, or optimize the corresponding part of a Transformer pipeline. It is also useful when a model API behaves unexpectedly and you need to trace the behavior back to tokens, tensor shapes, attention visibility, training, or inference mechanics.

When it is not enough

Understanding this mechanism does not by itself prove that a complete model or application is accurate, safe, fast, or cost-effective. Production decisions still require representative evaluation data, latency and memory measurements, model-specific documentation, and tests of the surrounding retrieval or agent code.

14. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: claiming decoder-only is universally the “best” architecture.

Why it is incorrect: As stated explicitly in Section 11, this isn’t accurate — it became dominant specifically for general-purpose LLM use cases, with real, acknowledged tradeoffs, not because it’s superior in every dimension for every task.

⚠️ Mistake

Incorrect idea: confusing cross-attention with self-attention.

Why it is incorrect: As demonstrated directly, they use the identical mathematical formula but draw Q, K, V from different sources — self-attention from one sequence, cross-attention from two.

⚠️ Mistake

Incorrect idea: assuming decoder-only models have “no encoder at all, anywhere in the pipeline.”

Why it is incorrect: They have no separate encoder architecture component — but the model still needs to process and “understand” the prompt, which it does via the same causal self-attention mechanism used for generation, just applied to the prompt tokens as part of the same unified sequence.

15. Important Distinctions

Self-AttentionCross-Attention
Q, K, V from the SAME sequenceQ from one sequence, K/V from a DIFFERENT sequence
Used in encoders, decoders, and decoder-only modelsUsed only in encoder-decoder models
Encoder-DecoderDecoder-Only
Separate encoder + decoder, connected via cross-attentionSingle unified sequence, causal self-attention only
Natural fit for tasks with a distinct source and targetNatural fit for general-purpose, open-ended generation

16. Production / Engineering Considerations

  • Decoder-only’s architectural simplicity (no separate encoder, no cross-attention) is part of why it scales relatively straightforwardly to very large sizes — one uniform block type, repeated, rather than two distinct sub-architectures to scale and balance.
  • Prompt engineering directly interacts with this architecture — since everything is one causally-processed sequence in a decoder-only model, the order and structure of what you put in a prompt genuinely affects what the model can “see” relative to what it’s currently generating.

17. Interview Questions

Beginner

Q: What is cross-attention, and how does it differ from self-attention?

Ans: Self-attention computes Query, Key, and Value all from the same sequence — a sequence relating to itself. Cross-attention computes the Query from one sequence (typically a decoder’s current state) while Key and Value come from a different sequence (typically an encoder’s output) — allowing one sequence to draw information directly from another.

Intermediate

Q: Where does cross-attention appear in an encoder-decoder architecture, and what problem does it solve?

Ans: It appears in each decoder block, alongside (but separate from) the decoder’s own causal self-attention. It solves the problem of letting the decoder access the full, complete source sequence’s information while generating each output token — without cross-attention, the decoder would have no direct mechanism for incorporating the encoder’s understanding of the source sequence into its generation process.

Advanced

Q: Why did decoder-only architectures become dominant for general-purpose LLMs, given that encoder-decoder architectures can also perform generation tasks?

Ans: A general-purpose LLM’s core training objective — next-token prediction — doesn’t have an inherent, structurally distinct “source” sequence requiring separate encoding; the prompt and the generated response can be treated as one continuous, causally-processed sequence. This avoids the architectural complexity of maintaining and training a separate encoder and cross-attention mechanism.

Additionally, decoder- only models trained at large scale on next-token prediction have empirically generalized well to a very wide range of tasks via prompting alone — tasks that might historically have used specialized encoder-only or encoder-decoder architectures — which reduced the practical need for task-specific architectural variants.

Scenario

Q: A team is deciding between a decoder-only and an encoder-decoder architecture for a document summarization system. What genuine tradeoffs would you raise?

Ans: Encoder-decoder offers an architecturally explicit separation between “fully understand the source document” (encoder, with bidirectional attention over the complete input) and “generate the summary” (decoder, with cross-attention pulling from that full understanding at each step) — a natural structural fit for this task’s clear source-and-target shape.

A decoder-only approach can also perform summarization effectively (treating the document and desired summary as one causal sequence, common in practice with modern general-purpose LLMs), with the practical advantage of using the same general-purpose model and infrastructure already in use for other tasks, rather than maintaining a specialized encoder-decoder model. The choice often comes down to whether the practical benefits of a single general-purpose decoder-only model outweigh the architectural fit of encoder-decoder for this specific, well-structured task.

Architecture

Q: Is it accurate to say a decoder-only model “has no encoder”?

Ans: It’s accurate in the sense that there’s no separate encoder architectural component with its own bidirectional attention and distinct parameters.

But the model still needs to build an understanding of the prompt — it does this using the same causal self-attention mechanism applied to the entire sequence (prompt plus generated tokens so far) uniformly, rather than through a dedicated, separately-designed encoding step.

Engineering

Q: Why does prompt ordering matter more in a decoder-only LLM than it might in an encoder-only model processing the same content?

Ans: In a decoder-only model, the entire input (including the prompt) is processed with causal self-attention (Module 6) — a given position can only attend to itself and earlier positions, never later ones.

This means information placed later in a prompt cannot influence how earlier information was “understood” during that same forward pass, unlike an encoder-only model’s full, bidirectional attention, where every position can consider the entire input regardless of order. This is a genuine, practical reason prompt structure and ordering matter for decoder-only LLM applications.

18. What You Should Remember

  • Cross-attention uses the same formula as self-attention, but draws Query from one sequence (the decoder) and Key/Value from a different sequence (the encoder) — verified directly with a real translation- style example.
  • Decoder-only became dominant for general-purpose LLMs specifically because next-token prediction doesn’t need a separate source sequence, and because decoder-only models generalize well via prompting — not because it’s a universally superior architecture in every dimension.
  • Encoder-only and encoder-decoder remain genuinely useful for tasks with a fixed, complete input (understanding) or a clear source-target structure (translation), respectively.

19. How This Helps Me Build AI Systems

Every LLM you’ll build agentic applications with is decoder-only — and you now understand precisely why, along with the honest tradeoffs involved, rather than a vague sense that “decoder-only won.” This directly explains why prompt structure and ordering matter so much when you’re assembling context for an LLM call.

Next: Module 14 — How GPT-Style LLMs Actually Work — tracing one complete request from prompt to generated token, integrating every mechanism from this course.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed