Decoder

The Encoder article covered the half of the Transformer built for deep, bidirectional understanding. This article covers the other half — the one that actually generates new text: the decoder.

The simple definition

A decoder is the part of a Transformer that generates output one token at a time, using everything generated so far plus (optionally) an encoder’s representation of some input, to predict what comes next. Recall from the Next-Token Prediction article that this is the exact, repeated task every modern chatbot performs. A decoder is the architectural component actually carrying out that task — and, importantly, most of today’s most prominent large language models, including every GPT model, use only a decoder, with no encoder half at all.

Why a decoder can only look backward, and why that’s essential

This is the single most important structural difference from the encoder, and it’s worth being precise about why it exists. Recall from the Encoder article that an encoder sees the entire input at once, in both directions. A decoder generating text can’t do this — at the moment it’s predicting the next token, the tokens that come after that point genuinely don’t exist yet. A decoder uses a specific technique called masked self-attention, which deliberately prevents each token from attending to any tokens that come after it in the sequence, only ever looking backward at what’s already been generated — exactly matching the real, physical constraint of next-token prediction described in the Next-Token Prediction article, where the model genuinely cannot see the future it hasn’t generated yet.

flowchart LR
    A["Generated so far: 'The sky is'"] --> B[Decoder: attends only to tokens already generated]
    B --> C["Predicts next: 'blue'"]
    C --> D["New sequence: 'The sky is blue'"]
    D --> E[Decoder attends to this full new sequence for the next prediction]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of an improv storyteller building a story live, one sentence at a time, in front of an audience. At any given moment, they can draw on everything they’ve already said — but they genuinely cannot draw on sentences they haven’t spoken yet, because those sentences don’t exist. Each new sentence has to make sense given only what’s already been said.

Where this breaks down: An improv storyteller has a sense of narrative direction and intent, even if loosely formed. A decoder’s “backward-only” constraint is a precise mathematical rule — masked self-attention, covered in the Self-Attention article — mechanically preventing any calculation from involving future tokens, not a storyteller’s intuitive discipline about not revealing plot points too early.

Why decoder-only models became the dominant architecture for chatbots

This directly connects to the “most LLMs use only a decoder” point made above. Recall from the Generative AI article that generation — producing new content, piece by piece — is the defining task of a chatbot. A decoder, purpose-built for exactly this backward-looking, one-token-at-a-time generation task, turns out to be sufficient on its own for this job, without needing a separate encoder half at all — the decoder simply treats the entire ongoing conversation, including the user’s messages, as the sequence it’s continuing. This architectural simplification — decoder-only, no separate encoder — is precisely what GPT (which stands for Generative Pre-trained Transformer) refers to in its own name.

A concrete example, layered

For a simple beginner example: a decoder generating the continuation of “Once upon a” can only see “Once,” “upon,” and “a” — it predicts “time” next based purely on that backward-looking context, then, once “time” is added, can see all four tokens when predicting the next one. For a production example: GPT-3’s architecture, as confirmed in OpenAI’s published paper, is a decoder-only Transformer with 96 layers — every one of GPT-3’s responses, and every response from GPT-4, Gemini, and Claude, is generated by a decoder repeatedly applying masked self-attention and next-token prediction, exactly the mechanism this article has described, at the scale of hundreds of billions of parameters.

Decoder masking with four positions

For The sky is blue, a causal decoder uses this visibility pattern:

Position	May attend to
The	The
sky	The, sky
is	The, sky, is
blue	The, sky, is, blue

The   ✓ · · ·
sky   ✓ ✓ · ·
is    ✓ ✓ ✓ ·
blue  ✓ ✓ ✓ ✓

It cannot use a future token to predict that same future token during training.

How GPT and Gemini use decoders

GPT’s published architectures are decoder-only Transformers. GPT-3’s 175B configuration stacks 96 decoder blocks and generates autoregressively.

Google’s Gemini report also describes enhanced Transformer decoders. Multimodal inputs become model representations, while text output is generated as a token sequence.

A decoder-only model can summarize, translate, classify, or answer questions by placing the task and relevant input in one causal context; it does not require a separate encoder for every task.

Trace two decoding steps

Suppose the prompt is “Water freezes at”.

Step 1 input:  Water freezes at
Visible tokens: Water, freezes, at
Likely next token selected: 0

Step 2 input:  Water freezes at 0
Visible tokens: Water, freezes, at, 0
Likely next token selected: °C

The decoder does not create the complete answer in one calculation. After selecting 0, it appends that token, runs another decoding step, and predicts again. The causal mask prevents the representation at an earlier position from reading future answer tokens during training.

Decoder-only does not mean input-free

GPT and Gemini can receive a long prompt, images, retrieved documents, tool results, and earlier messages. “Decoder-only” describes the main Transformer stack, not an application that has no input processing. All supplied information is converted into representations the decoder can attend to while generating its output.

For “Translate ‘good morning’ into French,” a decoder-only model treats the instruction and phrase as the prefix, then continues it with a likely answer such as “bonjour.” An original encoder-decoder Transformer would first encode the source separately, then let the decoder use cross-attention to read the encoder outputs.

Common misconception

A frequent beginner assumption: that a modern chatbot must use the full, original encoder-decoder Transformer architecture, since that’s the complete design the original 2017 paper described. As this article has explained, this is a genuine architectural simplification most large language models have made — GPT and similar models are decoder-only, having dropped the separate encoder half entirely, relying purely on the decoder’s own masked self-attention over the growing conversation sequence to handle both “understanding” the input and generating the output, all within one unified component.

Where this fits in what comes next

You now understand both halves of the original Transformer architecture, and why decoder-only designs became dominant for text generation. The next article, Attention, finally explains the actual mathematical mechanism — referenced constantly across both this article and the Encoder article — that lets a token “attend to” other tokens in the first place.

In one sentence

A decoder generates output one token at a time, using masked self-attention to look only backward at what’s already been generated — and decoder-only architectures, with no separate encoder half, are precisely what GPT, Gemini, and Claude are actually built from.

Related Terms

• Encoder
• Transformer
• Next-Token Prediction
• Attention
• Self-Attention