The Attention article mentioned, in passing, that Transformers run several attention calculations in parallel rather than just one. This article gives that detail the full treatment it deserves: multi-head attention.
The simple definition
Multi-head attention runs several independent attention calculations — called “heads” — in parallel on the same input, each with its own separately learned query, key, and value weight matrices, and then combines all their results together. Recall from the Attention article that a single attention calculation produces one specific way of weighing relevance across a sequence. Multi-head attention doesn’t rely on just one such calculation — it runs several, side by side, each free to learn a different kind of relevance pattern, then merges everything back into one combined, richer representation.
Why one single attention calculation isn’t enough
Recall from the Attention article’s “trophy and suitcase” example: resolving what “it” refers to is one specific kind of relationship. But a sentence contains many different kinds of relationships simultaneously — which words are grammatically linked (a verb to its subject), which words share a topic, which pronoun refers to which noun, which word modifies which other word. A single attention calculation, with one shared set of query, key, and value weights, would have to compress all of these genuinely different relationship types into one single, shared pattern of relevance — a real bottleneck. Multi-head attention solves this by giving each relationship type its own dedicated “head,” free to specialize.
flowchart LR
A[Input sequence] --> B[Head 1: learns grammatical relationships]
A --> C[Head 2: learns coreference, pronoun resolution]
A --> D[Head 3: learns topical relevance]
A --> E["...more heads"]
B --> F[Combine all heads' outputs]
C --> F
D --> F
E --> F
F --> G[Single, richer combined representation]ANALOGY vs. TECHNICAL REALITY
Analogy: Think of a manuscript being reviewed by several different specialist editors at once, rather than a single generalist editor trying to catch everything alone — one editor checks grammar, another checks factual consistency, another checks tone and style. Each editor focuses on their own specific concern, and their combined feedback, taken together, produces a far more thorough review than any single editor working alone could manage.
Where this breaks down: Human editors are assigned their specialty deliberately, by a manager who tells them what to look for. Attention heads aren’t assigned a specialty by anyone — each head starts with randomly initialized weights and, purely through the training process covered throughout the Training Mechanics phase, happens to settle into focusing on whatever kind of relationship turns out to reduce the loss function most effectively, without any human ever telling it “you handle pronouns.”
How the heads actually get combined back together
This is worth being concrete about, since “running several attention calculations” raises an obvious follow-up question: what happens to all those separate results afterward? Each head produces its own smaller output vector; these get concatenated (placed side by side) into one longer combined vector, which then passes through one final learned weight matrix that blends everything back down into a single, unified representation of the original size — ready to be passed to the next component in the layer. This final blending step is itself a learned transformation, meaning training also shapes how the different heads’ specialized perspectives get weighted and combined, not just what each head individually learns to focus on.
A concrete example, layered
For a simple beginner example: processing “The chef, who trained in Paris, cooked the meal quickly,” one attention head might specialize in linking “chef” to “cooked” (subject to verb, skipping over the interrupting clause), while a different head might specialize in linking “who” back to “chef” — two genuinely different relationships, each handled by its own dedicated head, rather than being blurred together into one compromised calculation. For a production example: GPT-3’s published architecture, as confirmed in OpenAI’s paper, uses 96 attention heads per layer, across 96 layers — meaning every single token gets its representation refined by 96 separate, specialized relevance calculations at every one of the 96 layers it passes through, a genuinely enormous amount of parallel, specialized relational analysis happening at every step.
Why more heads isn’t automatically better
It’s worth being honest about a real, practical trade-off here, echoing the “bigger isn’t automatically better” caution from the Large Language Model article. More attention heads mean more parameters and more computation per layer, directly increasing training and inference cost, as covered throughout the Training and Inference articles. Published research has also found that not every head ends up learning something distinct and useful — some heads, once training is complete, turn out to be somewhat redundant with others, contributing less than their computational cost might suggest, which is part of why head count is a genuine architectural design decision requiring real experimentation, not a setting where “more is obviously better.”
Common misconception
A frequent beginner assumption: that each attention head is deliberately assigned to track one specific, named grammatical role (like “the pronoun-resolution head” or “the verb-agreement head”) by the people who designed the architecture. As the Analogy section explained, this isn’t how it works — no human assigns heads their specialty; what each head ends up focusing on is an emergent, learned outcome of training, and researchers studying trained models afterward often do find heads that specialize in recognizable ways, but that specialization wasn’t designed in advance, only discovered after the fact.
Where this fits in what comes next
You now understand why Transformers run attention multiple times in parallel rather than just once. The next article, Feed-Forward Network, covers the other major component sitting inside every single Transformer layer — the piece that processes each token’s attention-refined representation individually, right after multi-head attention finishes its work.
In one sentence
Multi-head attention runs several independent attention calculations in parallel, each free to specialize in a different kind of relevance pattern, then combines their results into one richer representation — letting a Transformer track grammar, reference, and meaning simultaneously rather than compressing everything into a single, blurred calculation.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed