TechByteByByte

Transformers

Understand the Transformer architecture deeply enough to remove the black box — multi-head attention, residual connections, layer normalization, positional encoding, and why GPT-style LLMs are decoder-only — with verified numeric examples.

#Deep Learning#Neural Networks#AI#Transformers#Multi-Head Attention#LLMs

Begin with the central question

What architecture made it possible to process relationships across many tokens in parallel?

That question is the reason this topic exists. Keep it in mind as each new term appears: every equation, diagram, and code example below is one part of the answer.

tokens + positions → attention → feed-forward processing → repeated blocks

Before you continue: three tools for this module

  • Token: a piece of text represented by an ID.
  • Residual connection: a shortcut that adds a layer’s input back to its transformed output.
  • Feed-forward network: a small neural network applied independently to each token position inside a Transformer block.

You do not need to memorize these definitions yet. Use them as a small map whenever the terms appear below.

What You Will Understand

The complete Transformer block, assembled from everything Modules 1-15 built: multi-head attention, residual connections, layer normalization, positional encoding, and the feed-forward network — plus precisely why GPT-style LLMs specifically use a decoder-only architecture. This is the bridge module into understanding what an LLM actually is.

A Transformer block combines several responsibilities:

token vectors + position information
        ↓
masked self-attention → residual path → normalization
        ↓
position-wise feed-forward network → residual path → normalization
        ↓
next block

Exact ordering differs by architecture: modern models may use pre-normalization, rotary position information, gated MLPs, or other variants. “Transformer” names a family of architectures, not one frozen block diagram.

Why Attention Needs Position, Stable Depth, and Nonlinear Processing

Module 15 built self-attention from first principles. But self-attention alone has a gap: it has no inherent sense of token order (attention computes relevance regardless of position), and a single attention computation, stacked without other supporting structure, doesn’t train stably at depth.

The Transformer architecture wraps self-attention with exactly the additional components needed to fix both problems — and stacks the result into the deep networks (Module 10) that power every modern LLM.

Building the Car Around the Attention Engine

if Module 15 gave you the engine (attention), this module builds the whole car around it: positional information (so the engine knows where each token is, not just what it relates to), residual connections and normalization (so the whole thing can be built many layers deep without the training instability of Module 10), and a feed-forward network (extra nonlinear processing capacity at each position, independent of attention).

Analogy: The Highway Bypass & The Multi-Lens Committee

  • Residual Connections (The Highway Bypass): Imagine a town where all cars must drive down local streets, stop at every traffic light, and queue at every intersection (layers performing transformations). Driving across town takes hours, and the cars lose fuel (gradients decay to zero during backpropagation). A residual connection is like building an elevated highway overpass running directly over the town. Cars can choose to skip the local traffic and zip straight to the exit. Gradients use this bypass to travel directly back to early layers without decay.
  • Multi-Head Attention (The Committee with Special Lenses): Imagine a committee analyzing a text document. Instead of having one person read it, you hire 8 experts:
    • Expert 1 wears grammar glasses (focuses on subject-verb matches).
    • Expert 2 wears pronoun glasses (focuses on what “it” refers to).
    • Expert 3 wears timeline glasses (focuses on verb tenses and order).
  • They all read the text concurrently (in parallel heads) and compile their notes. Multi-head attention lets the model capture different semantic relationships simultaneously.

📊 Visual Diagram: Anatomy of a Transformer Decoder Block

Here is the block architecture containing residual highway lanes, self-attention, and position-wise feed-forward networks:

graph TD
    Input["Input Vectors (x)"] --> AddPos["Add Positional Encoding"]
    AddPos --> Split["Fork / Split Input"]

Split -->|Residual Highway Path| AddNorm1["Add and LayerNorm Node"]
    Split -->|Transformation Path| MultiHead["Multi-Head Attention<br>(Causal Masked)"]

MultiHead --> AddNorm1

AddNorm1 --> Split2["Fork Intermediate Output"]
    Split2 -->|Residual Highway Path| AddNorm2["Add and LayerNorm Node"]
    Split2 -->|Transformation Path| FFN["Feed-Forward Network<br>(Position-wise MLP)"]

FFN --> AddNorm2
    AddNorm2 --> BlockOutput["Block Output Vector"]

4. Core Concept

The complete flow through one Transformer block

Tokens
  ↓
Embeddings                    (Module 12)
  ↓
+ Positional Information        (this module — attention alone has
                                no sense of order)
  ↓
Self-Attention                  (Module 15 — often MULTI-HEAD)
  ↓
Residual connection + Normalization    (LayerNorm, Module 11)
  ↓
Feed-Forward Network             (a small 2-layer network, applied
                                 independently at each position)
  ↓
Residual connection + Normalization
  ↓
(repeat for the NEXT Transformer block)
  ↓
...
  ↓
Output

Multi-head attention

Instead of computing attention once, a Transformer computes it several times in parallel, each with its own separate W_q, W_k, W_v (“heads”), then combines the results:

head_1 = attention(X @ Wq_1, X @ Wk_1, X @ Wv_1)
head_2 = attention(X @ Wq_2, X @ Wk_2, X @ Wv_2)
...
multi_head_output = concatenate(head_1, head_2, ...) @ W_output

🧠 Why multiple heads? Each head can learn to focus on a different kind of relationship (e.g., one head might learn grammatical structure, another might learn topical similarity) — a single attention computation constrains all relevance-scoring into one shared subspace; multiple heads let the model represent several different kinds of relationships simultaneously.

Positional encoding

Self-attention (Module 15) is fundamentally order-agnostic — swapping two tokens’ positions in the input doesn’t change the attention computation’s relevance scores between them at all. Positional encoding adds information about where each token sits in the sequence directly into its embedding, before attention ever runs.

Residual connections

output = sublayer(x) + x

Instead of a layer’s output completely replacing its input, the original input is added back. This gives gradients a direct, short path backward through the network (an additional, powerful defense against vanishing gradients, Module 10) and lets very deep stacks of Transformer blocks train reliably.

5. How It Works — Step by Step

1. Convert input tokens into embeddings (Module 12)
2. Add positional encoding to each token's embedding
3. Compute MULTI-HEAD self-attention over the sequence
4. Add the ORIGINAL input back (residual connection),
   then apply LayerNorm (Module 11)
5. Pass the result through a feed-forward network (two linear
   layers with a nonlinear activation between them, Module 4 —
   applied independently to each position)
6. Add the residual again, apply LayerNorm again
7. This ENTIRE sequence (steps 3-6) is ONE Transformer block --
   modern LLMs stack many of these (commonly dozens)
8. The final block's output feeds into a final output layer
   (Module 17 covers exactly what that produces for an LLM)

6. Mathematical Intuition

First, use only small numbers

For three tokens, attention first lets each token collect relevant information from the others. The feed-forward stage then transforms each updated token separately. Stacking blocks repeats this two-part process while residual paths preserve useful earlier information.

Read the mathematics as a story

A Transformer alternates information mixing through attention with per-token transformation through feed-forward networks. Residual paths and normalization keep the repeated stack trainable.

tokens + positions → attention → feed-forward processing → repeated blocks

Do not begin by memorizing the symbols. First identify what enters, what operation changes it, and what comes out. The symbols are a compact description of that journey. Positional encoding, computed via alternating sine/cosine functions at different frequencies:

PE(position, 2i)   = sin(position / 10000^(2i/d_model))
PE(position, 2i+1) = cos(position / 10000^(2i/d_model))

position is the token’s index in the sequence; i indexes the embedding dimension; d_model is the total embedding dimension.

Each position gets a unique combination of sine/cosine values across dimensions, and — a deliberate mathematical property — the relative difference between two positions’ encodings has a mathematically consistent structure, letting the model learn to reason about relative, not just absolute, position.

Causal masking, the mechanism that prevents a decoder-only model from “cheating” by looking at future tokens:

Before masking: token at position i could attend to ALL positions,
                including ones that come AFTER it

After masking:  attention scores for any position j > i (a future
                position) are set to -infinity BEFORE softmax --
                softmax(-infinity) = 0, so those positions receive
                EXACTLY ZERO attention weight

7. Simple Example

Walk through the example

Read the example in three passes:

  1. Identify the input numbers and what each number represents.
  2. Follow one operation at a time instead of jumping directly to the answer.
  3. Interpret the final number in ordinary language and connect it back to the problem.

The purpose is not merely to calculate the result. It is to make the internal mechanism visible. Without positional encoding, the sentences “the dog bit the man” and “the man bit the dog” would produce identical self-attention relevance computations between “dog,” “bit,” and “man” — since attention alone only sees which tokens are present, not their order.

Positional encoding is what lets the model distinguish these two, otherwise identically-worded, opposite-meaning sentences.

8. Python Example

Three Python symbols used below

  • NumPy (np) is a Python library for working efficiently with lists and grids of numbers.
  • np.array(...) creates a numeric vector or matrix.
  • @ performs matrix multiplication: many connected weighted sums calculated together.

You can understand the concept without memorizing the syntax. First follow what the numbers represent, and then connect each code operation to the worked example.

What the code will demonstrate

Before running the code, predict the flow: create a small input, apply the topic’s calculation, and inspect the intermediate or final values. The example uses small numbers so you can connect each printed result to the explanation above; a real model performs the same kind of operation with much larger tensors and learned parameters.

# Build a tiny, inspectable example of Transformers.
# Follow the intermediate values; they reveal what the model is doing.
import numpy as np

def positional_encoding(seq_len, d_model):
    pos = np.arange(seq_len)[:, np.newaxis]
    i = np.arange(d_model)[np.newaxis, :]
    angle_rates = 1 / np.power(10000, (2 * (i // 2)) / np.float32(d_model))
    angles = pos * angle_rates
    pe = np.zeros((seq_len, d_model))
    pe[:, 0::2] = np.sin(angles[:, 0::2])
    pe[:, 1::2] = np.cos(angles[:, 1::2])
    return pe

pe = positional_encoding(seq_len=4, d_model=4)
print("Positional encoding (4 positions, 4 dims):\n", np.round(pe, 4))

token_embeddings = np.array([
    [1.0, 0.0, 1.0, 0.0],
    [0.0, 1.0, 0.0, 1.0],
    [1.0, 1.0, 0.0, 0.0],
    [0.5, 0.5, 0.5, 0.5],
])
combined = token_embeddings + pe
print("\nToken embeddings + positional encoding:\n", np.round(combined, 4))

Expected Output:

Positional encoding (4 positions, 4 dims):
 [[ 0.      1.      0.      1.    ]
 [ 0.8415  0.5403  0.01    1.    ]
 [ 0.9093 -0.4161  0.02    0.9998]
 [ 0.1411 -0.99    0.03    0.9996]]

Token embeddings + positional encoding:
 [[ 1.      1.      1.      1.    ]
 [ 0.8415  1.5403  0.01    2.    ]
 [ 1.9093  0.5839  0.02    0.9998]
 [ 0.6411 -0.49    0.53    1.4996]]

Causal masking, verified — showing exactly why a decoder-only model cannot see the future:

# Build a tiny, inspectable example of Transformers.
# Follow the intermediate values; they reveal what the model is doing.
def softmax(x, axis=-1):
    exp_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return exp_x / np.sum(exp_x, axis=axis, keepdims=True)

seq_len = 4
scores = np.random.RandomState(1).randn(seq_len, seq_len)
print("Raw attention scores (before masking):\n", np.round(scores, 3))

causal_mask = np.triu(np.ones((seq_len, seq_len)), k=1).astype(bool)
masked_scores = scores.copy()
masked_scores[causal_mask] = -np.inf
print("\nCausal-masked scores (future positions set to -inf):\n", masked_scores)

masked_attention = softmax(masked_scores, axis=-1)
print("\nAttention weights after causal masking:\n", np.round(masked_attention, 4))

Expected Output:

Raw attention scores (before masking):
 [[ 1.624 -0.612 -0.528 -1.073]
 [ 0.865 -2.302  1.745 -0.761]
 [ 0.319 -0.249  1.462 -2.06 ]
 [-0.322 -0.384  1.134 -1.1  ]]

Causal-masked scores (future positions set to -inf):
 [[ 1.62434536        -inf        -inf        -inf]
 [ 0.86540763 -2.3015387         -inf        -inf]
 [ 0.3190391  -0.24937038  1.46210794        -inf]
 [-0.3224172  -0.38405435  1.13376944 -1.09989127]]

Attention weights after causal masking:
 [[1.     0.     0.     0.    ]
 [0.9596 0.0404 0.     0.    ]
 [0.2126 0.1204 0.6669 0.    ]
 [0.1495 0.1406 0.6413 0.0687]]

9. How It Works

  • Position 0’s attention weights are [1, 0, 0, 0] — it can only attend to itself, since every later position was masked to -inf before softmax. Position 3 (the last, in this 4-token example) has non-zero weight across all four positions — it can see everything up to and including itself. This precisely demonstrates causal masking: each position can only attend to itself and earlier positions, never later ones.
  • A residual connection (x + sublayer_output) keeps the original input’s values clearly present in the output rather than being fully replaced — the sublayer adjusts the representation rather than overwriting it, exactly the “direct gradient path” intuition from Section 4.

10. Encoder, Decoder, and Decoder-Only

Encoder:            processes an ENTIRE input sequence at once,
                    with FULL (non-causal) self-attention -- every
                    position can see every other position, useful
                    for understanding a complete input (e.g., a
                    sentence to be translated)

Decoder:            generates output ONE TOKEN AT A TIME, using
                    CAUSAL self-attention (demonstrated above) so
                    it can never "cheat" by seeing tokens it hasn't
                    generated yet; often also uses CROSS-attention
                    to incorporate an encoder's output (Module 15)

Encoder-Decoder:     the original Transformer design (e.g., for
                    translation) -- an encoder processes the input
                    language, a decoder generates the output
                    language, using cross-attention to connect them

Decoder-only:        uses ONLY the decoder side, with causal
                    self-attention, no separate encoder and no
                    cross-attention -- generates text by
                    repeatedly predicting the next token, using
                    only what it has generated (or been given) so far

⚠️ Do not confuse “decoder-only” with the traditional encoder-decoder architecture. GPT-style LLMs are decoder-only — there is no separate encoder, and no cross-attention. Every attention computation in a decoder-only model is self-attention (Module 15), with causal masking. This is architecturally simpler than the original encoder-decoder Transformer, not a variant that still includes an encoder.

Why GPT-style LLMs are decoder-only: their core task — predicting the next token given everything so far — is naturally and entirely a “generate one token at a time, only looking backward” problem.

There’s no separate “source” sequence to encode (unlike translation, where a distinct input language needs encoding before generating the output language) — decoder-only architecture is the natural, sufficient fit for pure next-token-prediction language modeling.

11. How Is This Used in Modern AI?

Follow it from mechanism to product

GPT-style LLMs use decoder-oriented Transformer stacks; other Transformer systems may use encoders or encoder-decoder designs. The architecture produces model outputs, while chat history management, retrieval, tools, and safety controls belong to the surrounding application.

How this connects to LLMs

prompt → tokens → deep-learning computations → next-token probabilities → generated response

The model computation is only the middle of the journey. Tokenization happens before it, while decoding and application controls happen afterward; the following example identifies this topic’s exact role.

🤖 Real-world connection

This entire module is modern AI’s dominant architecture. GPT-style LLMs (including Claude) are decoder-only Transformers, stacked dozens to over a hundred blocks deep, each block containing exactly the multi-head-attention → residual/norm → feed-forward → residual/norm sequence described here.

ConceptAI application
Multi-head or related attention variantsContext mixing in Transformer-based language models
Causal maskingWhat makes autoregressive (next-token) generation work correctly
Positional encoding (or learned variants)How every Transformer-based model represents token order
Decoder-only architectureGPT, Claude, and most modern general-purpose LLMs
Encoder-decoderStill used for some translation and certain specialized sequence-to-sequence tasks

12. How Is This Used in Agentic AI?

Trace one agent step

goal + history + tool results → LLM proposal → runtime validation → tool or response

The deep-learning model produces a prediction or structured proposal. The agent runtime—ordinary software around the model—controls permissions, executes tools, stores state, handles retries, and decides whether another model call is needed.

Direct relevance to Agentic AI: Very High. The Transformer architecture, specifically decoder-only, causally-masked, is what every agent’s underlying LLM actually is.

Understanding this architecture is what makes an agent’s behavior — its ability to maintain context across a long conversation, reason over retrieved documents, and generate coherent multi-step plans — a traceable consequence of specific mechanisms (self-attention, residual depth, causal generation) rather than an unexplainable black box.

13. Common Beginner Mistakes / Misconceptions Corrected

⚠️ Mistake

Incorrect idea: “Transformers are just attention.”

Why it is incorrect: As stated in Module 15 and reiterated here — a full Transformer block includes multi-head attention, residual connections, layer normalization, AND a feed-forward network. Attention is the defining new mechanism, but not the entire architecture.

⚠️ Mistake

Incorrect idea: decoder-only models still have a hidden encoder component.

Why it is incorrect: They don’t (Section 10) — decoder-only means exactly that: no separate encoder, no cross-attention, purely causal self-attention throughout.

⚠️ Mistake

Incorrect idea: positional encoding and token embeddings are the same thing.

Why it is incorrect: They’re added together (Section 8’s combined example) but represent different information: token embeddings encode what a token is; positional encoding encodes where it sits in the sequence.

14. Important Distinctions

EncoderDecoder
Full (non-causal) self-attention — sees the whole input at onceCausal self-attention — can only see itself and earlier positions
Good for understanding a complete inputGood for generating output token by token
Self-AttentionCross-Attention (Module 15)
Q, K, V all from the same sequenceQ from one sequence, K/V from another (e.g., decoder attending to encoder output)
Encoder-DecoderDecoder-Only
Separate encoder + decoder, connected via cross-attentionOnly a decoder, purely causal self-attention, no cross-attention
Classic translation-style architectureGPT-style LLMs, including Claude

15. When to Use

Decoder-only Transformers are the standard choice for general-purpose language modeling and modern LLMs. Encoder-decoder architectures remain relevant for tasks with a clear, distinct source-to-target structure (like translation) or certain specialized sequence-to-sequence tasks.

16. When Not to Use

For very short, simple sequence tasks, a full Transformer’s depth and complexity may be unnecessary overhead compared to a simpler model — this is a genuine engineering trade-off, not a universal rule that Transformers are always the right choice regardless of task scale.

17. Interview Questions

Beginner

Q: What are the main components of a single Transformer block?

Ans: Multi-head self-attention, a residual connection followed by layer normalization, a feed-forward network, and another residual connection followed by layer normalization — this whole sequence is one block, and modern LLMs stack many of these.

Intermediate

Q: Why is positional encoding necessary, given that attention already processes the whole sequence?

Ans: Self-attention computes relevance scores between tokens based purely on their content (via Query/Key similarity) — it has no inherent notion of token order, so swapping two tokens’ positions wouldn’t change their computed relevance to each other at all.

Positional encoding adds information about each token’s position directly into its embedding before attention runs, which is the only way the model can distinguish sequences that use the same words in different orders.

Advanced

Q: Explain precisely why GPT-style LLMs are decoder-only, and what “decoder-only” specifically excludes.

Ans: GPT-style LLMs are trained purely to predict the next token given everything so far — there’s no separate “source” sequence requiring encoding, unlike tasks such as translation, which have a distinct source and target sequence connected via cross-attention. Decoder-only specifically excludes: a separate encoder component, and cross-attention of any kind.

Every attention computation in a decoder-only model is self-attention over the sequence generated (or provided) so far, using causal masking to prevent any position from attending to positions that come after it — verified directly in this module, where position 0’s attention weights were entirely zero for every later position.

Scenario

Q: You’re told a new model uses “full bidirectional attention with no causal masking.” Would this be suitable for autoregressive text generation like GPT? Why or why not?

Ans: No — without causal masking, every position could attend to every other position, including ones that come later in the sequence.

For autoregressive generation, where the model predicts each next token based only on what’s come before, this would mean the model could “see” the very token it’s supposed to be predicting during training, making the task trivial and useless for generation (though full bidirectional attention is exactly appropriate for encoder-style tasks, like understanding a complete, already-available input sequence).

AI Engineering

Q: When you send a long conversation history to Claude or a similar LLM, what is architecturally happening, in terms of this module’s concepts?

Ans: The entire conversation (system prompt, prior turns, and the new message) is tokenized, embedded, and combined with positional encoding, then passed through many stacked decoder-only Transformer blocks — each applying causal self-attention (so each token can only attend to itself and everything before it), residual connections, normalization, and a feed-forward network.

The final block’s output is used to predict a probability distribution over the next token (Module 17 covers this final step precisely), which is generated, appended to the sequence, and the whole process repeats for each subsequent token.

18. What You Should Remember

  • A Transformer block = multi-head self-attention + residual connections/LayerNorm + feed-forward network + residual connections/LayerNorm, stacked many times.
  • Positional encoding is what lets an otherwise order-agnostic attention mechanism understand sequence order.
  • Causal masking — verified directly, with position 0 able to attend only to itself — is what makes decoder-only, autoregressive generation possible.
  • GPT-style LLMs are decoder-only: no separate encoder, no cross-attention, purely causal self-attention throughout.

19. How This Helps Me Build AI Systems

You now understand, mechanism by mechanism, what happens inside the architecture every LLM you’ll ever call is built from. Module 17 completes the picture: tracing a full input all the way to a generated next token, using exactly the components you’ve now built and verified, piece by piece, since Module 1.

Next: Module 17 — How LLMs Actually Use Deep Learning — the complete trace from raw text to next-token prediction, integrating everything from this course.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed