The Feed-Forward Network article covered the second major computation inside a Transformer layer. This article covers something wrapped around both that and multi-head attention — not a computation exactly, but a structural shortcut: the residual connection.
The simple definition
A residual connection adds a sub-layer’s original input directly to its output, creating a shortcut path that skips around whatever transformation the sub-layer performed. Instead of a layer’s output being simply output = SubLayer(input), a residual connection makes it output = input + SubLayer(input) — the original input passes straight through unchanged, alongside whatever the sub-layer actually computed, and the two get added together.
flowchart LR
A[Input to sub-layer] --> B[Sub-layer: e.g., Multi-Head Attention]
A --> D["Shortcut: input passes through unchanged"]
B --> C[Sub-layer's output]
C --> E[Add together]
D --> E
E --> F[Final output: input + sub-layer's contribution]Why this genuinely necessary fix exists: the vanishing gradient problem, revisited
Recall directly from the Backpropagation article’s discussion of vanishing gradients: in very deep networks, gradients calculated for the earliest layers can shrink to almost nothing by the time they’ve been chained backward through many layers, meaning those early layers barely get updated at all during training. Recall also from the ReLU article that ReLU helped address part of this problem for individual activation functions — but stacking many full layers deep, as GPT-3’s 96 layers do, reintroduces a related, structural version of the same problem. Residual connections directly address this: because the shortcut path lets a gradient flow straight through, unmodified, during backpropagation, it gives every layer — even ones very early in a 96-layer-deep network — a direct, largely undiminished path back to the loss, rather than forcing that gradient signal to survive being multiplied through every single intermediate layer’s calculation.
ANALOGY vs. TECHNICAL REALITY
Analogy: Think of a relay race where, alongside each runner passing the baton forward in the usual way, there’s also a direct shortcut lane letting a copy of the baton skip straight ahead to the finish line, bypassing any one specific runner who might stumble. Even if one runner in the middle of the race performs badly, the shortcut lane ensures the baton still reaches the end reliably, rather than the whole race depending on every single runner performing flawlessly.
Where this breaks down: A relay race’s baton is a single physical object being handed off. A residual connection’s “shortcut” is a mathematical addition operation, applied to entire vectors, at every single sub-layer — the original input isn’t literally bypassing computation to “arrive first,” it’s being combined, through simple addition, with whatever transformation the sub-layer produced, at every layer, throughout the whole depth of the network simultaneously.
Why this specific fix, rather than just training more carefully
It’s worth being direct about how significant this technique actually is, since it’s easy to describe as a small implementation detail when it’s genuinely foundational. Residual connections were introduced by He and colleagues in a landmark 2015 paper studying image-recognition networks, and they demonstrated that networks using this technique could be trained successfully at depths that were previously impractical — with the technique becoming standard practice across deep learning generally soon afterward, and adopted directly into the original 2017 Transformer architecture. Without residual connections, stacking 96 Transformer layers deep, as GPT-3 does, would likely be an exercise in diminishing, unstable returns rather than the genuinely productive depth that makes large language models capable of what they do.
A concrete example, layered
For a simple beginner example: in a small 4-layer Transformer, a residual connection at layer 2 ensures that even if layer 2’s attention calculation contributes only a small, subtle refinement to a token’s representation, the token’s original, layer-1 information isn’t lost or overwritten — it’s preserved and carried forward, with layer 2’s contribution added on top rather than replacing it entirely. For a production example: GPT-3’s 96-layer architecture, as confirmed in OpenAI’s published paper, wraps both the multi-head attention and feed-forward network sub-layers in every single one of its 96 layers with residual connections — without this structural choice, training a model this deep, at this scale, would likely have been dramatically harder, if not practically infeasible, using the gradient-based training methods covered throughout the Training Mechanics phase.
Common misconception
A frequent beginner assumption: that residual connections are a minor optimization trick, useful but not essential — something a model could technically do without, just less efficiently. As the vanishing gradient discussion above explained, this understates their real importance — for genuinely deep architectures like modern Transformers, residual connections aren’t a nice-to-have efficiency improvement, they’re close to a hard requirement for training to succeed at all, a foundational structural choice rather than an optional tuning knob.
Where this fits in what comes next
You now understand the shortcut structure that makes deep Transformer stacking trainable. The next article, Layer Normalization, covers the other essential stabilization technique paired directly alongside residual connections in every single Transformer sub-layer.
In one sentence
A residual connection adds a sub-layer’s original input directly to its output, creating a shortcut that lets both information and gradients flow through largely undiminished — the specific, foundational structural fix that makes training genuinely deep Transformer stacks, like GPT-3’s 96 layers, practically possible at all.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed