TechByteByByte

The Transformer Block

Assemble multi-head attention, residual connections, layer normalization, and the feed-forward network into the complete, repeating unit every Transformer-based LLM is built from — with a fully verified end-to-end computation.

#Transformers#Transformer Block#Residual Connections#LayerNorm#AI#LLM

Begin with the central question

What happens inside one reusable Transformer block?

Essential words

A Transformer block combines attention, a feed-forward network, residual paths, and normalization. A sublayer is one major computation inside the block. Blocks are stacked repeatedly.

What You Will Understand

How Modules 3-8’s individual pieces — attention, multi-head attention, positional information — assemble with residual connections and layer normalization into one complete Transformer block: the exact unit stacked dozens of times to build a real LLM. You’ll trace a full block computation end to end, with real numbers.

states -> attention -> residual/normalization -> FFN -> residual/normalization

The problem this module solves

Every component so far has been examined in isolation. But attention alone, stacked many times with nothing else, doesn’t train reliably at depth — you already know why from the Deep Learning course (DL Module 10-11): vanishing gradients and unstable activations. The Transformer block exists to wrap attention with exactly the supporting structure needed to make deep stacking actually work.

Build the intuition

if attention is the engine, the Transformer block is the complete drivetrain — residual connections give the engine’s power a reliable path through many layers (DL Module 10-11’s fix for vanishing gradients), normalization keeps values well-behaved at every stage, and the feed-forward network gives each token extra processing capacity independent of its relationship to other tokens.

Analogy: The Engine Drivetrain & The Residual Bypass Highway Think of a complete Transformer block in terms of automotive design and highway navigation:

  • The Attention Engine: Attention is a high-performance engine that communicates details across tokens. But an engine alone doesn’t move a car.
  • The Residual Highway (The Bypass): Deep stacking causes vanishing gradients (the engine stalls out). Residual connections act as a direct, toll-free express bypass lane running parallel to the slow city streets (the attention math). Gradients skip the toll booths and travel backward directly to early layers.
  • The FFN Processing Desk: The Feed-Forward Network gives each token a private workspace to process updates independently without talking to other tokens (independent transformation post-communication).
  • The Layer Norm Shocks: Normalization functions like a suspension system, smoothing out bumpy fluctuations in vector scales at every stage.

📊 Visual Flowchart: Pre-LN Transformer Block Layout

Here is the structural wiring of a standard pre-LN block (used in modern LLMs):

graph TD
    X["Input X (Sequence Length x d_model)"] --> LN1["1. LayerNorm 1"]
    LN1 --> MHA["2. Multi-Head Attention"]




    X --> Add1["3. Residual Add: X + Attention(LN1)"]
    MHA --> Add1




    Add1 --> LN2["4. LayerNorm 2"]
    LN2 --> FFN["5. Feed-Forward Network (Position-wise FFN)"]




    Add1 --> Add2["6. Residual Add: Output1 + FFN(LN2)"]
    FFN --> Add2




    Add2 --> BlockOut["7. Block Output Matrix (Seq Length x d_model)"]

4. Core Concept

Input
  ↓
Multi-Head Attention          (Module 7)
  ↓
Residual Connection             (add the ORIGINAL input back)
  ↓
Layer Normalization              (DL Module 11 — reused here, not re-derived)
  ↓
Feed Forward Network               (Module 10 of this course)
  ↓
Residual Connection                  (add the input to THIS sub-layer back)
  ↓
Layer Normalization
  ↓
Output

⚠️ A Transformer is not “just attention.” As this diagram makes explicit, one block contains attention AND two residual connections AND two normalization steps AND a feed-forward network. Attention is the architecturally novel piece — but the full block is what actually trains reliably and performs well.

Two genuinely different jobs happen inside one block:

Attention  =  COMMUNICATION between tokens
              (each token gathers information FROM other tokens)




FFN        =  TRANSFORMATION of each token's OWN representation
              (independently, with no communication between tokens
              — Module 10 covers this precisely)

5. How It Works — Step by Step

1. Input X enters the block (embeddings + positional info, or the
   previous block's output)
2. Multi-head attention (Module 7) computes a new, contextual
   representation for each token
3. RESIDUAL: add the ORIGINAL input X back to the attention
   output -- output = X + attention(X)
4. LAYER NORM: normalize the result
5. Feed the normalized result through the FEED-FORWARD NETWORK
   (Module 10) -- applied independently to each token position
6. RESIDUAL: add the input to THIS step back -- output =
   norm_1_output + FFN(norm_1_output)
7. LAYER NORM: normalize again
8. This is the COMPLETE output of one Transformer block -- the
   SAME SHAPE as the original input, ready to be fed into the
   NEXT block (or, if this is the final block, into the output
   layer -- Module 14)

6. Mathematical Intuition

Read the mathematics as a story

A block preserves tensor shape while changing information. Attention mixes across token positions; the FFN transforms each position; residual paths and normalization keep the stack trainable.

X -> attention -> add/norm -> FFN -> add/norm -> same shape, richer values

Residual connections, precisely: output = x + sublayer(x). The critical property (DL Module 10-11, worth restating in this specific context): during backpropagation, this addition gives gradients a direct path back to x, unmodified by sublayer’s potentially-shrinking derivatives — across many stacked Transformer blocks, this is what prevents the vanishing gradient problem from recurring at Transformer depth.

7. Small Worked Example

Walk through the example

  1. Begin with a (tokens, d_model) matrix. 2. Run attention. 3. Add the input and normalize. 4. Run the FFN. 5. Add and normalize again.

Given a 3-token sequence with d_model=4, one block’s computation preserves the (3, 4) shape throughout every step — attention output is (3, 4), added residually to a (3, 4) input stays (3, 4), normalized stays (3, 4), and even though the feed-forward network’s internal hidden layer expands to a larger dimension (d_ff, commonly 4x d_model in real models), its final output projects back down to (3, 4) before the second residual connection — this shape consistency is what allows blocks to be stacked arbitrarily deep.

8. Python / NumPy Example

What the code will demonstrate

This small NumPy example makes The Transformer Block visible with inspectable numbers and shapes. Read it in three passes: identify each input, follow the transformation line by line, and connect the printed output to the diagram above. The arrays are intentionally tiny teaching values; unless the text explicitly says otherwise, they are not weights or measurements from GPT, Gemini, or another trained model.

# The arrays are intentionally small so each transformation can be inspected.
# Printed values illustrate the mechanism; they are not trained-model measurements.
import numpy as np




def softmax(x, axis=-1):
    exp_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return exp_x / np.sum(exp_x, axis=axis, keepdims=True)




def layer_norm(x, eps=1e-8):
    mean = x.mean(axis=-1, keepdims=True)
    std = x.std(axis=-1, keepdims=True)
    return (x - mean) / (std + eps)




def relu(x):
    return np.maximum(0, x)




np.random.seed(10)




seq_len = 3
d_model = 4
d_ff = 8   # feed-forward inner dimension (commonly larger than d_model)




X = np.round(np.random.randn(seq_len, d_model) * 0.5, 3)
print("Input X (embeddings + positional info):\n", X)




# --- Multi-head attention (single head at full d_model, for clarity) ---
Wq = np.round(np.random.randn(d_model, d_model) * 0.4, 2)
Wk = np.round(np.random.randn(d_model, d_model) * 0.4, 2)
Wv = np.round(np.random.randn(d_model, d_model) * 0.4, 2)
Wo = np.round(np.random.randn(d_model, d_model) * 0.4, 2)




Q, K, V = X @ Wq, X @ Wk, X @ Wv
scores = Q @ K.T / np.sqrt(d_model)
weights = softmax(scores, axis=-1)
attn_out = (weights @ V) @ Wo
print("\nAttention output:\n", np.round(attn_out, 4))




# --- Residual + LayerNorm #1 ---
residual_1 = X + attn_out
norm_1 = layer_norm(residual_1)
print("\nAfter residual + LayerNorm #1:\n", np.round(norm_1, 4))
print("Per-token mean (should be ~0):", np.round(norm_1.mean(axis=-1), 6))
print("Per-token std (should be ~1):", np.round(norm_1.std(axis=-1), 6))




# --- Feed-forward network ---
W1 = np.round(np.random.randn(d_model, d_ff) * 0.4, 2)
b1 = np.zeros(d_ff)
W2 = np.round(np.random.randn(d_ff, d_model) * 0.4, 2)
b2 = np.zeros(d_model)




ffn_hidden = relu(norm_1 @ W1 + b1)
ffn_out = ffn_hidden @ W2 + b2
print("\nFFN output:\n", np.round(ffn_out, 4))




# --- Residual + LayerNorm #2 ---
residual_2 = norm_1 + ffn_out
norm_2 = layer_norm(residual_2)
print("\nAfter residual + LayerNorm #2 (Transformer block OUTPUT):\n", np.round(norm_2, 4))
print("Per-token mean (should be ~0):", np.round(norm_2.mean(axis=-1), 6))
print("Per-token std (should be ~1):", np.round(norm_2.std(axis=-1), 6))




print("\nInput shape:", X.shape, "-> Output shape:", norm_2.shape, "(UNCHANGED)")

Expected Output:

Input X (embeddings + positional info):
 [[ 0.666  0.358 -0.773 -0.004]
 [ 0.311 -0.36   0.133  0.054]
 [ 0.002 -0.087  0.217  0.602]]




Attention output:
 [[ 0.0025 -0.0263  0.0496 -0.0689]
 [ 0.0042 -0.0223  0.0435 -0.0615]
 [ 0.0045 -0.0213  0.0419 -0.0598]]




After residual + LayerNorm #1:
 [[ 1.1912  0.5414 -1.4936 -0.239 ]
 [ 1.1068 -1.5577  0.5769 -0.126 ]
 [-0.6725 -1.1314  0.3359  1.4679]]
Per-token mean (should be ~0): [-0.  0.  0.]
Per-token std (should be ~1): [1. 1. 1.]




FFN output:
 [[ 1.9638  0.1528  1.2474  0.4519]
 [ 3.1725  0.1888 -0.1246 -1.0979]
 [ 0.2807 -0.3287  0.4014  0.3484]]




After residual + LayerNorm #2 (Transformer block OUTPUT):
 [[ 1.6756 -0.1978 -0.9137 -0.5641]
 [ 1.6443 -0.8359 -0.0362 -0.7722]
 [-0.4629 -1.3349  0.4586  1.3392]]
Per-token mean (should be ~0): [0. 0. 0.]
Per-token std (should be ~1): [1. 1. 1.]




Input shape: (3, 4) -> Output shape: (3, 4) (UNCHANGED)

9. How It Works

  • Every intermediate result keeps the (3, 4) shape — attention output, both residual sums, both normalized outputs, and the final block output all match the original input’s shape exactly, confirmed directly in the printed shapes.
  • After each LayerNorm step, every token’s mean is essentially 0 and standard deviation is essentially 1 — LayerNorm genuinely doing its job (DL Module 11), verified numerically rather than assumed.
  • The FFN’s internal hidden layer expands to d_ff=8 (double d_model here; real models often use 4x or more) before projecting back down to d_model=4 — this expand-then-contract pattern is standard, giving the network extra intermediate capacity per token (Module 10 covers this precisely).
  • The final output’s shape matching the input’s shape is exactly what allows this block to be stacked repeatedly — the output of block 1 can become the input to block 2 with zero shape mismatch, all the way through however many blocks the model has.

10. How Is This Used in Modern AI?

Where this concept lives

Follow the concept at three levels: inside the model, where the computation happens; inside the AI product, where that computation supports a visible feature; and inside production, where engineers measure speed, memory, quality, and failure cases. The details below connect those levels.

🤖 How Is This Used in Modern AI?

This is not a simplified teaching abstraction — this exact sequence (attention → residual → norm → FFN → residual → norm) is, structurally, what runs inside every real Transformer block in every modern LLM, just with far larger dimensions, multi-head attention with many heads (Module 7), and often gated FFN variants (Module 10).

11. How Is This Used in LLMs?

Trace one model call

User text → tokens → Transformer computation → output-token probabilities
          this topic affects one part of that computation

An LLM does not apply this idea as a separate magic step. It uses it as part of the repeated numerical pipeline that transforms token vectors and produces the next-token probabilities.

A real LLM stacks this exact block dozens to over a hundred times (Module 12 covers the full stacked architecture). Each successive block receives the previous block’s output as its input, building increasingly rich, contextual representations — layer by layer, exactly the “hierarchy of representations” intuition from the Deep Learning course, now made concrete for the Transformer architecture specifically.

Real systems you can recognize

Hugging Face model outputs can expose hidden states after embeddings and after each layer; see model outputs. Production models vary in pre-norm versus post-norm order and in FFN design.

12. How Is This Used in Agentic AI?

Trace one agent step

Goal + history + tool results
              ↓
     LLM processes the context
              ↓
Suggested answer or tool call
              ↓
Agent runtime validates and executes it

This distinction matters: the Transformer helps produce the proposal, while the surrounding agent software controls tools, permissions, retries, memory, and execution.

Direct relevance to Agentic AI: High, entirely through the underlying LLM. Every reasoning capability an agent’s LLM demonstrates — tracking context, relating a tool result back to the original request — is built from many repetitions of exactly this block.

There is no separate “reasoning module” inside an LLM; there’s this block, stacked many times, and the final representation it produces (Module 14 traces this to a generated token).

When this knowledge is useful

Use The Transformer Block when you need to explain, implement, debug, evaluate, or optimize the corresponding part of a Transformer pipeline. It is also useful when a model API behaves unexpectedly and you need to trace the behavior back to tokens, tensor shapes, attention visibility, training, or inference mechanics.

When it is not enough

Understanding this mechanism does not by itself prove that a complete model or application is accurate, safe, fast, or cost-effective. Production decisions still require representative evaluation data, latency and memory measurements, model-specific documentation, and tests of the surrounding retrieval or agent code.

13. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: thinking “Transformer” and “attention” are synonyms.

Why it is incorrect: Restated deliberately from Section 4 — a full block includes residual connections, normalization, and a feed-forward network, not just attention.

⚠️ Mistake

Incorrect idea: assuming the FFN operates across tokens, like attention does.

Why it is incorrect: It doesn’t — Module 10 covers this precisely, but the key point now: FFN is applied independently, per token position, with zero communication between positions. Attention is the only place information moves between different tokens within a block.

⚠️ Mistake

Incorrect idea: forgetting residual connections apply around EACH sub-layer separately.

Why it is incorrect: There are two residual connections per block — one around attention, one around the FFN — not one single residual wrapping the entire block.

14. Important Distinctions

Attention (within a block)Feed-Forward Network (within a block)
Communication BETWEEN tokensTransformation of EACH token independently
Depends on the whole sequenceDepends only on that one token’s own representation
Residual ConnectionLayer Normalization
Adds the sub-layer’s input back to its outputRescales values to consistent mean/std
Primarily helps GRADIENT FLOW across depthPrimarily helps TRAINING STABILITY

15. Production / Engineering Considerations

  • Pre-LN vs. Post-LN (where exactly normalization sits relative to the residual connection) is a real architectural variation across different model families, covered specifically in Module 11 — this module used one common ordering for clarity.
  • Number of stacked blocks (model “depth”) is one of the primary levers determining a model’s total parameter count and capability, alongside width (d_model) and number of attention heads.

16. Interview Questions

Beginner

Q: What are the main components of a single Transformer block?

Ans: Multi-head attention, a residual connection, layer normalization, a feed-forward network, another residual connection, and another layer normalization — in that sequence. This entire sequence is one block, and real models stack many of these.

Intermediate

Q: What’s the fundamental difference between what attention does and what the feed-forward network does, within a Transformer block?

Ans: Attention is how tokens communicate — each token’s new representation incorporates information gathered from other tokens in the sequence, weighted by relevance.

The feed-forward network is applied independently to each token’s own representation, with no communication between different token positions at all — it transforms each token’s information in isolation, using the same learned weights across all positions.

Advanced

Q: Why does a Transformer block need TWO separate residual connections rather than one wrapping the entire block?

Ans: Each sub-layer (attention, and separately the FFN) benefits from its own direct gradient path back to its own input — wrapping only the whole block would still leave a potentially-vanishing-gradient path through both sub-layers internally.

Applying residual connections around each sub-layer individually, as demonstrated in this module’s step-by-step computation, ensures gradients have a short, direct path around every individual transformation, not just around the block as a whole.

Scenario

Q: You’re inspecting a Transformer block’s output and find that the per-token mean and standard deviation aren’t close to 0 and 1 respectively, right after a LayerNorm step. What would you suspect?

Ans: LayerNorm is specifically designed to normalize each token’s values to approximately mean 0 and standard deviation 1 (verified directly in this module’s output) — if that’s not the case immediately after a LayerNorm computation, I’d suspect a bug in the LayerNorm implementation itself (e.g., normalizing across the wrong axis, or a numerical epsilon issue), rather than treating it as expected variation.

Architecture

Q: If you removed the feed-forward network from every Transformer block, keeping only attention, residuals, and normalization, what capability would the model lose?

Ans: The model would lose its capacity for independent, per-token transformation beyond what attention’s weighted combination of other tokens’ values provides. Attention can only produce new representations as combinations of existing tokens’ Value vectors — it doesn’t have its own independent nonlinear transformation capacity the way the FFN does.

Removing the FFN would significantly limit the model’s overall representational power, since a substantial portion of a Transformer’s total parameters and processing capacity typically lives in the FFN sub-layers, not attention.

Engineering

Q: Why does it matter, from an engineering perspective, that a Transformer block’s output shape exactly matches its input shape?

Ans: This shape consistency is what allows blocks to be stacked arbitrarily deep — block 2 can take block 1’s output directly as its input with zero reshaping or adaptation needed, and this repeats for however many blocks the model has. Without this consistent shape contract, stacking blocks would require additional reshaping logic between every layer, adding real architectural complexity for no benefit.

AI Engineering

Q: When someone says a model has “32 layers,” what does that concretely mean in terms of what you learned in this module?

Ans: It means the model stacks 32 complete Transformer blocks — each one containing its own multi-head attention, residual connections, layer normalization, and feed-forward network, with its own independently learned parameters.

The input passes through all 32 blocks in sequence, each one refining the token representations further, before the final block’s output is used to produce the model’s actual prediction (Module 14 traces this final step).

17. What You Should Remember

  • A Transformer block is **attention + residual + norm + FFN + residual
    • norm** — verified end to end, with shapes and LayerNorm statistics confirmed at every stage.
  • Attention = communication between tokens. FFN = transformation of each token independently. This distinction is fundamental.
  • The block’s output shape exactly matches its input shape — the property that allows arbitrary stacking depth.

18. How This Helps Me Build AI Systems

You’ve now assembled and verified the complete, exact repeating unit every real LLM is built from — not a simplified stand-in, but the genuine architecture, just at toy scale. Module 12 stacks this block into the full model; nothing conceptually new needs to be introduced there.

Next: Module 10 — Feed Forward Networks — a deeper look at what happens after attention, including modern gated variants like SwiGLU.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed