Both the Encoder and Decoder articles referenced tokens “attending to” other tokens without explaining exactly how. It’s time to open up that mechanism directly: attention.
The simple definition
Attention is a mechanism that lets a model calculate how much each token in a sequence should influence its understanding of every other token. Rather than treating every other word as equally relevant, attention lets a model learn — through training, exactly as covered throughout the Training Mechanics phase — to weigh some tokens far more heavily than others when interpreting a given word, based on genuine, learned relevance rather than simple proximity or fixed rules.
Why this idea was such a breakthrough
Recall from the Encoder article’s “bank by the river” example: understanding a word correctly often depends on specific other words elsewhere in a sentence, not necessarily the words directly next to it. Before attention, models generally struggled to directly connect distant, relevant words without that connection weakening the further apart they were, since older architectures had to pass information step by step through every word in between, an easy path for relevant signal to fade along the way. Attention solves this by letting any token connect directly to any other token, regardless of distance, with the strength of that connection learned from data rather than fixed by physical proximity.
flowchart LR
A["'it'"] -->|high attention| B["'animal' (what 'it' refers to)"]
A -->|low attention| C["'the', 'was', 'too'"]How attention actually calculates “relevance”: queries, keys, and values
This is the real, mathematical heart of the mechanism, and it’s worth walking through concretely rather than staying purely conceptual. For every token, the model calculates three separate vectors — recall from the Vector article that a vector is just an ordered list of numbers — called a query, a key, and a value, each produced by multiplying the token’s representation against its own learned weight matrix (recall the weight matrices from the Weights article). The query represents “what am I looking for”; the key represents “what do I offer”; the value represents “what information do I actually contribute if I’m relevant.” Attention then compares each token’s query against every other token’s key — using the dot product covered in the Dot Product article — to calculate a relevance score for every pair, converts those scores into a proper probability distribution using softmax (exactly as covered in the Probability Distribution article), and finally uses those probabilities to calculate a weighted blend of every token’s value — producing a new, context-enriched representation for the original token.
flowchart LR
A[Token's Query] --> C[Compare against every token's Key via dot product]
B[Every token's Key] --> C
C --> D[Softmax: convert scores to attention weights]
D --> E[Weighted blend of every token's Value]
E --> F[New, context-enriched representation]ANALOGY vs. TECHNICAL REALITY
Analogy: Think of a research librarian helping you find relevant sources for a specific question. Your question (the query) gets compared against the subject tags on every book in the library (the keys), and books whose tags closely match your question get pulled and weighted more heavily in the answer you’re given — their actual content (the values) contributing far more to your final answer than books whose tags barely matched.
Where this breaks down: A librarian applies genuine judgment about relevance and quality. The query-key-value comparison is a precise, mechanical dot-product calculation, applied identically at every single position in every layer — no judgment, just learned weight matrices producing numbers that happen to reflect genuinely useful relevance patterns after training, as covered throughout the Training Mechanics phase.
Multi-head attention: doing this comparison in several different ways at once
It’s worth naming a real, practical detail here, since it shows up directly in published model architectures like GPT-3’s. Rather than calculating attention just once, Transformers typically use multi-head attention — running several independent attention calculations in parallel, each with its own separately learned query, key, and value weight matrices, covered in full in its own dedicated article right after Self-Attention. Different attention “heads” often end up specializing in different kinds of relationships — one head might focus on grammatical relationships (subject to verb), another on coreference (a pronoun to what it refers to) — giving the model several simultaneous, complementary “lenses” through which to weigh relevance, rather than relying on just one.
A concrete example, layered
For a simple beginner example: in the sentence “The trophy didn’t fit in the suitcase because it was too big,” attention lets the model calculate a strong connection between “it” and “trophy” (not “suitcase”), correctly resolving the ambiguous pronoun by weighing “trophy” far more heavily in “it“‘s context-enriched representation — a genuinely hard problem for older architectures, and a textbook example used repeatedly in attention research. For a production example: GPT-3’s published architecture uses 96 attention heads per layer, across 96 layers, as confirmed in OpenAI’s paper — meaning every single token, at every layer, gets its representation refined through 96 separate, parallel relevance calculations, repeated 96 times over as information flows deeper through the network.
Calculate one attention head with small vectors
Suppose the current token has query q = [1, 0]. Three tokens have keys:
The → [0, 1]
cat → [1, 0]
sat → [0.5, 0.5]Dot-product scores are:
q · key(The) = 0
q · key(cat) = 1
q · key(sat) = 0.5Softmax turns [0, 1, 0.5] into approximate attention weights:
The = 18.6%
cat = 50.6%
sat = 30.7%The weighted value vectors are then added to produce a new contextual representation. Real attention also scales scores by the square root of the key dimension before softmax.
flowchart LR
A[Query] --> D[Query-key dot products]
B[Keys] --> D
D --> E[Scale and softmax]
E --> F[Attention weights]
C[Values] --> G[Weighted sum]
F --> G
G --> H[Contextual output]Real GPT and Gemini attention data
The original Transformer base model used 8 attention heads. OpenAI’s published GPT-3 175B configuration used 96 heads across its 96-layer decoder-only Transformer.
The Gemini 1.0 report describes efficient attention mechanisms including multi-query attention, where heads can share key/value representations to reduce decoding memory and bandwidth.
A second example: attention can mix information
Consider “The animal did not cross the road because it was tired.” When updating the representation for it, an attention head may place more weight on animal than on road. The value vector from animal then contributes more strongly to the new representation for it.
Illustrative attention weights for the query “it”
animal ████████ 0.62
road ██ 0.14
tired ██ 0.16
other █ 0.08
----
1.00These numbers are illustrative, not weights read from GPT or Gemini. Real models calculate different attention patterns at every layer and head. Attention weights also should not automatically be treated as a perfect explanation of why the complete model produced an answer; later layers and feed-forward networks continue changing the representation.
What multiple heads can learn to notice
Using several heads lets the layer form several weighted mixtures in parallel. In one sentence, different heads may become useful for nearby grammar, long-distance references, sentence boundaries, code structure, or other learned relationships. Engineers do not normally assign “pronoun head” or “code head” labels in advance—the patterns emerge during training and can overlap.
GPT-3’s published 175-billion-parameter configuration used 96 attention heads in each of 96 layers. That does not mean it performed only 96 comparisons. For every layer, every token position participates in attention calculations across the sequence and across those heads.
Common misconception
A frequent beginner assumption: that “attention” means something like human attention or focus — the model consciously deciding what’s important, the way a person might. As the query-key-value explanation above made clear, attention is a precise mathematical calculation — dot products, softmax, weighted averaging — with no consciousness or deliberate focus involved, just learned weight matrices producing numbers that happen to correlate remarkably well with what a human would consider genuinely relevant, after enough training on enough data.
Where this fits in what comes next
You now understand the general mechanism. The next article, Self-Attention, covers the specific, most common variant of this mechanism — where a sequence attends to itself, rather than to some separate, different sequence — which is exactly what powers the decoder-only architecture covered in the previous article.
In one sentence
Attention calculates, through learned queries, keys, and values compared via dot products and softmax, exactly how much each token should influence a model’s understanding of every other token — the precise mathematical mechanism that finally explains everything the Encoder and Decoder articles described only conceptually.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed