The Multi-Head Attention article covered the component responsible for tokens exchanging information with each other. This article covers what happens immediately afterward, inside every single Transformer layer: the feed-forward network.
The simple definition
A feed-forward network (FFN) is a small, simple neural network — two linear layers with an activation function in between — applied to each token’s representation individually, right after multi-head attention. Recall from the Neural Network article’s discussion of layers and nodes: a feed-forward network here is genuinely just that same basic structure from the Neural Networks phase, applied at a specific point inside a Transformer layer, processing one token’s vector at a time.
Why attention alone isn’t the whole story
This is worth being precise about, since it directly clarifies what each piece of the Transformer actually contributes. Recall from the Attention article that attention’s whole job is letting tokens exchange information with each other — mixing context across the sequence. What attention does not do is apply any further, deep, non-linear processing to a token’s own representation once that mixing is done. The feed-forward network fills exactly this gap: after attention has gathered relevant context from across the sequence into each token’s representation, the feed-forward network takes that enriched representation and processes it further, independently, giving the model additional computational depth and the chance to transform that mixed information into something more useful.
flowchart LR
A[Multi-Head Attention: tokens exchange information with each other] --> B[Feed-Forward Network: each token's representation processed individually]
B --> C[Passed to the next Transformer layer]What “position-wise” actually means, concretely
This is a genuinely important, specific detail worth being clear about. The feed-forward network is applied position-wise — meaning the exact same small network, with the exact same weights, gets applied separately to every single token’s vector in the sequence, one at a time, with no mixing between tokens happening at this step at all. Recall from the Node article’s weighted-sum-plus-bias-plus-activation calculation: the feed-forward network is essentially two of these calculations stacked together — expand the token’s vector to a larger size, apply an activation function (commonly ReLU or GELU, as covered in the ReLU article), then compress it back down to its original size.
flowchart LR
A["Token 1's vector"] --> D[Same FFN weights applied]
B["Token 2's vector"] --> E[Same FFN weights applied]
C["Token 3's vector"] --> F[Same FFN weights applied]
D --> G["Token 1's transformed vector"]
E --> H["Token 2's transformed vector"]
F --> I["Token 3's transformed vector"]ANALOGY vs. TECHNICAL REALITY
Analogy: Think of a group discussion (multi-head attention) where everyone shares information and updates their understanding based on what others said, followed by each person quietly stepping away to think through their own updated notes individually, refining their own conclusions privately before the group reconvenes for the next round of discussion.
Where this breaks down: A person’s private reflection draws on genuine, flexible reasoning. The feed-forward network’s “private processing” is a fixed, identical mathematical transformation — the exact same two weight matrices and activation function, applied without any variation, to every single token, at every single position, with zero awareness of what any other specific token’s content actually is at that point in the calculation.
Why the feed-forward network typically expands, then compresses
This is a real, well-documented architectural detail worth naming, since it explains why this component has so many of a model’s total parameters. The first linear layer typically expands a token’s vector to a larger intermediate size — often four times wider, as in the original 2017 Transformer paper’s specification — before the second linear layer compresses it back down to the original size. This expand-then-compress pattern gives the network more room to represent complex transformations in that intermediate, wider space, and it’s a major reason the feed-forward network sub-layer typically accounts for the majority of a Transformer’s total parameter count, more than the attention mechanism itself.
A concrete example, layered
For a simple beginner example: after multi-head attention has enriched the word “bank” in “the bank by the river” with context confirming it means riverbank rather than a financial institution, the feed-forward network takes that specific, now-disambiguated vector and applies its own learned transformation to it — potentially sharpening or further refining features relevant to “riverbank” specifically, independent of whatever is happening to any other word in the sentence at that same moment. For a production example: GPT-3’s published architecture, as confirmed in OpenAI’s paper, uses a hidden dimension of 12,288 for its main representations, with its feed-forward networks expanding to roughly four times that width internally before compressing back down — a real, substantial computational step repeated independently for every token, at every one of GPT-3’s 96 layers.
Common misconception
A frequent beginner assumption: that the feed-forward network is a minor, secondary detail compared to the “real” work attention does. As the parameter-count discussion above showed, this significantly understates its role — the feed-forward network typically holds the majority of a Transformer’s total parameters, and its position-wise, non-linear processing is doing genuinely substantial computational work, not a small afterthought tacked onto the more famous attention mechanism.
Where this fits in what comes next
You now understand both major computational components inside a Transformer layer — attention (tokens exchanging information) and the feed-forward network (each token processed individually). The next article, Residual Connections, covers a structural technique wrapped around both of these components, essential for making it possible to stack many such layers together without training collapsing.
In one sentence
The feed-forward network is a simple, two-step neural network applied identically and independently to every token’s representation, right after multi-head attention has finished mixing context between tokens — and it typically holds the majority of a Transformer’s parameters, doing far more computational work than its modest name might suggest.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed