Begin with the central question
After tokens exchange information through attention, why must each token pass through another neural network?
Essential words
The FFN applies the same small neural network independently at every position. Its intermediate dimension is commonly wider than model width. A nonlinear activation lets it transform features rather than only remix them linearly.
What You Will Understand
What happens after attention inside a Transformer block: the position-wise feed-forward network. You’ll verify, directly and numerically, that it transforms each token completely independently — zero communication between positions — and cover modern gated variants (GLU, GeGLU, SwiGLU) at an intuitive level.
contextual token state -> expand -> activation/gating -> project backThe problem this module solves
Module 9 established that attention handles communication between tokens, while the FFN handles transformation of each token’s own representation. This module goes deeper into exactly why that second piece exists: attention’s output is a weighted combination of existing Value vectors — it has no independent capacity to apply new, nonlinear transformations. The FFN supplies exactly that missing capacity.
Build the intuition
after attention gathers relevant information from across the sequence into each token’s representation, the FFN gives each token a private moment to process what it just gathered — like a group discussion (attention) followed by everyone individually writing down their own conclusions (FFN), with no further discussion happening during the writing.
4. Real-World Analogy
Think of attention as a group meeting where everyone shares relevant updates, and the FFN as each person then going back to their own desk to independently process what they heard and update their own notes.
The meeting (attention) involves everyone talking to everyone; the independent note-updating (FFN) happens at each person’s desk, completely separately, using the same “note-taking method” (the same learned weights) for everyone, but with no further cross-talk.
Analogy: The Private Office Desks & SwiGLU Desk Drawers Think of processing information post-meeting in an office cubicle structure:
- The Private Desk (The standard FFN): After the big conference room discussion (attention), everyone walks back to their own separate cubicles (token positions). Each person takes their notes, expands them into a full page (Linear expansion
d_ff), reviews them with a non-linear highlight checklist (ReLU/GELU activation), and condenses them back into a single summary bullet point (Linear contractiond_model).- No cross-talk happens between cubicles. It is completely parallelized.
- SwiGLU Gated Cubicle Drawers: Instead of just a highlighting checklist, you have two drawers:
- Drawer A (Gate Projection): Decides which subjects are relevant to remember today.
- Drawer B (Value Projection): Holds the raw notes.
- You run Drawer A through a dimming switch (SiLU activation) and multiply it elementwise by the contents of Drawer B. If the gate drawer says 0% for a topic, that topic is deleted from the notes.
📊 Visual Flowchart: SwiGLU Gated Feed-Forward Network
Here is the math pipeline for the gated linear unit (GLU) variants used in LLaMA models:
graph TD
X["Input Vector: x (post-attention)<br>(Dimension: d_model)"] --> GateProj["1. Gate Projection<br>(x @ W_gate)"]
X --> ValueProj["1. Value Projection<br>(x @ W_value)"]
GateProj --> SiLU["2. Gate Activation<br>(SiLU / Swish)"]
subgraph GatingOperation ["Gated Linear Activation"]
SiLU --> GateMultiply["3. Elementwise Multiply: gate * value"]
ValueProj --> GateMultiply
end
GateMultiply --> OutProj["4. Contraction Projection: W_out<br>(Map back to d_model)"]
OutProj --> FfnOut["Gated FFN Output (Dimension: d_model)"]5. Core Concept
x (one token's representation, post-attention and post-norm)
↓
Linear (expansion: d_model -> d_ff, typically d_ff = 4 × d_model)
↓
Activation (nonlinearity — DL Module 4)
↓
Linear (contraction: d_ff -> d_model)
↓
Output (same shape as input)| Term | Definition |
|---|---|
| Position-wise FFN | A feed-forward network applied identically and independently to every token position |
| Dimensional expansion | The first linear layer maps up to a larger inner dimension (d_ff) |
| Dimensional reduction | The second linear layer maps back down to d_model |
| GLU (Gated Linear Unit) | A variant using two projections, one “gating” the other via elementwise multiplication |
| SwiGLU | A GLU variant using the SiLU/Swish activation on the gate — common in modern LLMs |
6. How It Works — Step by Step
Standard FFN:
1. Take a token's representation (dimension d_model)
2. Project UP to a larger dimension d_ff (commonly 4x d_model)
via a learned linear layer
3. Apply a nonlinear activation function (DL Module 4 — ReLU,
GELU, or similar)
4. Project back DOWN to d_model via a second learned linear layer
5. This happens IDENTICALLY, but INDEPENDENTLY, for every token
position in the sequence -- the SAME weights are reused for
every position, but no position's computation involves any
OTHER position's dataSwiGLU-style gated FFN:
1. Take a token's representation
2. Compute TWO separate linear projections: a "gate" and a "value"
3. Apply SiLU/Swish activation to the GATE projection only
4. Multiply the activated gate ELEMENTWISE with the value
projection -- this is the "gating" mechanism: the gate
controls how much of each value dimension passes through
5. Project the gated result down to d_model7. Mathematical Intuition
Read the mathematics as a story
The FFN is applied to each token independently with shared weights. It usually expands the vector, applies a nonlinearity or gate, and projects it back.
one token d_model -> wider hidden vector -> activation/gate -> d_model
repeat same network for every tokenThe gating mechanism, conceptually: gated_hidden = SiLU(x @ W_gate) * (x @ W_value) — the * here is elementwise multiplication, not matrix multiplication. Each dimension of value gets scaled by its corresponding dimension of the activated gate — dimensions where the gate is near 0 get suppressed; dimensions where the gate is large pass through more strongly.
This gives the network a learned, per-dimension “how much of this should pass through” mechanism, beyond what a single activation function alone provides.
8. Small Worked Example
Walk through the example
- Select one token row. 2. Expand it. 3. Apply the activation. 4. Project it back. 5. Verify other token rows use the same weights.
If token A and token B both pass through the same FFN weights, token A’s output depends only on token A’s input — never on token B’s, even though they’re processed with identical weights. This is fundamentally different from attention, where every token’s output depends on information gathered from every other token.
The proof below demonstrates this precisely: shuffling the order of tokens before the FFN and comparing to shuffling the FFN’s output afterward produces identical results either way.
9. Python / NumPy Example
What the code will demonstrate
This small NumPy example makes Feed Forward Networks visible with inspectable numbers and shapes. Read it in three passes: identify each input, follow the transformation line by line, and connect the printed output to the diagram above. The arrays are intentionally tiny teaching values; unless the text explicitly says otherwise, they are not weights or measurements from GPT, Gemini, or another trained model.
# The arrays are intentionally small so each transformation can be inspected.
# Printed values illustrate the mechanism; they are not trained-model measurements.
import numpy as np
def relu(x): return np.maximum(0, x)
def sigmoid(x): return 1 / (1 + np.exp(-x))
def silu(x): return x * sigmoid(x) # SiLU/Swish, used in SwiGLU
np.random.seed(20)
seq_len = 3
d_model = 4
d_ff = 8
X = np.round(np.random.randn(seq_len, d_model) * 0.5, 3)
print("Input to FFN:\n", X)
# --- Standard FFN: Linear -> Activation -> Linear ---
W1 = np.round(np.random.randn(d_model, d_ff) * 0.4, 2)
b1 = np.zeros(d_ff)
W2 = np.round(np.random.randn(d_ff, d_model) * 0.4, 2)
b2 = np.zeros(d_model)
hidden = relu(X @ W1 + b1)
standard_ffn_out = hidden @ W2 + b2
print("\nStandard FFN output:\n", np.round(standard_ffn_out, 4))
print("Hidden layer shape:", hidden.shape, "(expanded from d_model=4 to d_ff=8)")
# --- SwiGLU-style gated FFN ---
W_gate = np.round(np.random.randn(d_model, d_ff) * 0.4, 2)
W_value = np.round(np.random.randn(d_model, d_ff) * 0.4, 2)
W_out = np.round(np.random.randn(d_ff, d_model) * 0.4, 2)
gate = silu(X @ W_gate)
value = X @ W_value
gated_hidden = gate * value # elementwise -- the "GLU" mechanism
swiglu_out = gated_hidden @ W_out
print("\nSwiGLU-style FFN output:\n", np.round(swiglu_out, 4))
# --- Prove: FFN transforms each token INDEPENDENTLY ---
X_shuffled = X[[2, 0, 1]] # reorder the token rows
hidden_shuffled = relu(X_shuffled @ W1 + b1)
out_shuffled = hidden_shuffled @ W2 + b2
print("\nOriginal FFN output:\n", np.round(standard_ffn_out, 4))
print("\nFFN output on SHUFFLED input (rows reordered 2,0,1):\n", np.round(out_shuffled, 4))
print("\nDoes shuffled output == original output rows reordered the same way?",
np.allclose(out_shuffled, standard_ffn_out[[2, 0, 1]]))Expected Output:
Input to FFN:
[[ 0.442 0.098 0.179 -1.172]
[-0.542 0.28 0.47 -0.489]
[ 0.252 0.203 0.162 -0.247]]
Standard FFN output:
[[ 0.2767 0.1594 0.0245 -0.4973]
[-0.1029 0.0936 0.0566 0.1202]
[ 0.1185 -0.0499 0.126 -0.0284]]
Hidden layer shape: (3, 8) (expanded from d_model=4 to d_ff=8)
SwiGLU-style FFN output:
[[-0.0572 0.0243 -0.0047 -0.1714]
[-0.0263 -0.0143 -0.0194 0.0495]
[-0.0015 -0.0021 0.0017 0.0016]]
Original FFN output:
[[ 0.2767 0.1594 0.0245 -0.4973]
[-0.1029 0.0936 0.0566 0.1202]
[ 0.1185 -0.0499 0.126 -0.0284]]
FFN output on SHUFFLED input (rows reordered 2,0,1):
[[ 0.1185 -0.0499 0.126 -0.0284]
[ 0.2767 0.1594 0.0245 -0.4973]
[-0.1029 0.0936 0.0566 0.1202]]
Does shuffled output == original output rows reordered the same way? True10. How It Works
- The standard FFN expands from
d_model=4tod_ff=8(a 4x expansion here, matching common real-model ratios), applies ReLU, then contracts back to4— exactly the expand-activate-contract pattern from Section 5. - The SwiGLU variant uses two separate projections (
gateandvalue) instead of one, gated via elementwise multiplication — a structurally different, more expressive mechanism than a single activation function, at the cost of an extra learned projection matrix. - The independence proof is the key result: reordering the input
tokens before the FFN, versus reordering the output after the FFN,
produces identical results (
True) — direct, numerical confirmation that the FFN never mixes information across token positions. Compare this to Module 8’s attention proof, where reordering did change results — the FFN’s behavior here is the deliberate opposite.
11. How Is This Used in Modern AI?
Where this concept lives
Follow the concept at three levels: inside the model, where the computation happens; inside the AI product, where that computation supports a visible feature; and inside production, where engineers measure speed, memory, quality, and failure cases. The details below connect those levels.
🤖 How Is This Used in Modern AI?
The FFN sub-layer typically contains the majority of a Transformer’s total parameters — with
d_ffcommonly 4x (or more) larger thand_model, and two large weight matrices per FFN, this sub-layer’s parameter count often dwarfs the attention sub-layer’s. Most modern LLMs (LLaMA-family and many others) use SwiGLU or similar gated variants rather than a plain ReLU FFN, based on empirical performance improvements found through experimentation.
12. How Is This Used in LLMs?
Trace one model call
User text → tokens → Transformer computation → output-token probabilities
this topic affects one part of that computationAn LLM does not apply this idea as a separate magic step. It uses it as part of the repeated numerical pipeline that transforms token vectors and produces the next-token probabilities.
Since the FFN operates identically on every token position with shared weights, it’s often described as where a Transformer stores much of its “factual” or “pattern” knowledge — some interpretability research suggests FFN layers function similarly to a large lookup/associative memory, though — consistent with this course’s caution against overclaiming interpretability — this remains an active research area, not a settled, universally-agreed mechanism.
Real systems you can recognize
The Transformer paper pairs attention with position-wise feed-forward networks. Current models often use gated variants such as SwiGLU rather than the exact original ReLU FFN, so architecture details vary.
13. How Is This Used in Agentic AI?
Trace one agent step
Goal + history + tool results
↓
LLM processes the context
↓
Suggested answer or tool call
↓
Agent runtime validates and executes itThis distinction matters: the Transformer helps produce the proposal, while the surrounding agent software controls tools, permissions, retries, memory, and execution.
Direct relevance to Agentic AI: Moderate, indirectly. The FFN’s role — independent per-token transformation using learned, general patterns — contributes to the underlying LLM’s ability to apply consistent “knowledge” or reasoning patterns to each token’s context, regardless of position, which underlies an agent’s ability to reason consistently within a single forward pass.
When this knowledge is useful
Use Feed Forward Networks when you need to explain, implement, debug, evaluate, or optimize the corresponding part of a Transformer pipeline. It is also useful when a model API behaves unexpectedly and you need to trace the behavior back to tokens, tensor shapes, attention visibility, training, or inference mechanics.
When it is not enough
Understanding this mechanism does not by itself prove that a complete model or application is accurate, safe, fast, or cost-effective. Production decisions still require representative evaluation data, latency and memory measurements, model-specific documentation, and tests of the surrounding retrieval or agent code.
14. Common Beginner Mistakes
⚠️ Mistake
Incorrect idea: assuming the FFN somehow relates different tokens to each other.
Why it is incorrect: As proven directly in this module, it does not — its output for any token depends only on that token’s own input, verified by the shuffle-invariance test.
⚠️ Mistake
Incorrect idea: thinking gated variants like SwiGLU are a completely different mechanism from a standard FFN.
Why it is incorrect: They’re a refinement — two projections and elementwise gating instead of one projection and a plain activation — but the same expand/transform/contract structure and the same per-token independence apply to both.
⚠️ Mistake
Incorrect idea: underestimating the FFN’s share of a model’s total parameters.
Why it is incorrect: As noted in Section 11, it’s frequently the largest single component, not a minor addition after “the real work” of attention.
15. Important Distinctions
| Attention | Feed-Forward Network |
|---|---|
| Output for a token depends on ALL tokens | Output for a token depends ONLY on that token — verified directly |
| No learnable per-position specialization (same Q/K/V logic applied uniformly) | Same weights applied to every position, but the actual computation for each token is fully self-contained |
| Standard FFN (ReLU) | Gated FFN (SwiGLU) |
|---|---|
| One projection up, one activation, one projection down | Two projections (gate + value), elementwise gating, one projection down |
Simpler, fewer parameters for the same d_ff | More expressive, more parameters, common in modern LLMs |
16. Production / Engineering Considerations
d_ffsize is a real architectural trade-off — larger inner dimensions increase model capacity and parameter count (and therefore compute/memory cost) without changingd_modelor sequence length directly.- Gated variants (SwiGLU etc.) add an extra weight matrix compared
to a standard FFN at the same
d_ff— a genuine parameter/compute cost trade-off against their empirically observed performance benefit.
17. Interview Questions
Beginner
Q: What does the feed-forward network inside a Transformer block do?
Ans: It applies a learned, nonlinear transformation to each token’s representation independently — typically expanding to a larger dimension, applying an activation function, then projecting back down to the original dimension. It uses the same weights for every token position, but processes each position’s data completely separately, with no interaction between positions.
Intermediate
Q: How is the FFN’s role different from attention’s role within a Transformer block?
Ans: Attention allows tokens to communicate — each token’s new representation is a weighted combination of information gathered from other tokens in the sequence. The FFN processes each token’s representation in complete isolation — its output for any given token depends only on that token’s own input, never on any other token’s data, even though the same learned weights are applied at every position.
Advanced
Q: How does a SwiGLU-style gated FFN differ mechanically from a standard ReLU-based FFN?
Ans: A standard FFN uses one linear projection up, one activation function, and one linear projection down. A SwiGLU FFN instead computes two separate linear projections of the input — a “gate” and a “value” — applies the SiLU/Swish activation to the gate only, then combines the two via elementwise multiplication (the value scaled by the activated gate) before a final projection down.
This gating mechanism gives the network a learned, per-dimension control over how much information passes through, which is a more expressive mechanism than a single shared activation function, at the cost of an additional weight matrix.
Scenario
Q: You’re told a bug caused two tokens’ positions to accidentally swap somewhere between the FFN’s input and output. Based on this module, would you expect this to affect the model’s final output meaningfully, and why?
Ans: Yes, meaningfully — while the FFN itself processes each token independently (so swapping wouldn’t corrupt the actual FFN computation for either token individually), the token at position 3 receiving what should have been position 5’s representation (or vice versa) means the WRONG content is now associated with each position going forward.
Since later layers (including subsequent attention computations) rely on positional information being correctly associated with the right content, this kind of bug would likely produce confused or incorrect downstream reasoning, even though the FFN computation itself remained “correct” for whatever input it happened to receive.
Architecture
Q: Why does the FFN typically contain more parameters than the attention sub-layer within the same Transformer block?
Ans: The FFN expands to a much larger inner dimension (d_ff, commonly 4x d_model or more) with two full weight matrices at that scale (expansion and contraction), while attention’s Q/K/V/output projections typically operate at or near d_model dimension.
This size difference means the FFN sub-layer often accounts for the majority of a Transformer block’s — and therefore the overall model’s — total parameter count.
Engineering
Q: Why might a team choose a gated FFN variant like SwiGLU over a
standard ReLU FFN, given it requires more parameters for the same
d_ff?
Ans: Because empirical results across many modern LLM training runs have shown gated variants like SwiGLU tend to produce better model quality for a given parameter/compute budget compared to standard FFNs — this is largely an empirically-driven architectural choice adopted by many current model families, not something derivable purely from theory, which is why this course treats it as a practical, evidence-based design decision rather than a mathematically “obvious” improvement.
AI Engineering
Q: If someone claims “the FFN layers are where an LLM stores its factual knowledge,” how would you respond, given this module’s guidance?
Ans: I’d note that some interpretability research does suggest FFN layers function somewhat like an associative memory or pattern-lookup mechanism, which is a genuinely interesting and actively studied hypothesis — but I’d be careful not to state this as a fully settled, proven fact.
Consistent with avoiding overclaiming interpretability (also flagged when discussing individual attention heads in Module 7), it’s more accurate to describe this as an active research area with supporting evidence, not a definitively established mechanism.
18. What You Should Remember
- The FFN is a position-wise, expand-activate-contract network, applied with shared weights but computed completely independently per token — proven directly: shuffling input tokens produces the exact same shuffle in the output.
- SwiGLU and other gated variants use two projections and elementwise gating instead of one projection and a plain activation — the modern common choice in real LLMs.
- The FFN sub-layer often holds the majority of a Transformer’s total parameters.
19. How This Helps Me Build AI Systems
You now understand precisely what happens to each token’s information after attention gathers context from the rest of the sequence — and can correctly explain why this step, unlike attention, is where per-token, position-independent processing happens. This distinction (Module 9’s “communication vs. transformation”) is a genuinely common, practical interview and architecture-reading topic.
Next: Module 11 — Residual Connections and Layer Normalization — a focused, Transformer-specific look at these two components, including Pre-LN vs. Post-LN.
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed