Attention: How an LLM Decides Which Words Matter Right Now

One word cannot understand a sentence by looking only at itself

Read this sentence:

The animal did not cross the street because it was tired.

What does the word “it” refer to?

Most people understand that “it” probably refers to the animal.

Now change the ending:

The animal did not cross the street because it was flooded.

This time, “it” probably refers to the street.

The word it did not change.

The earlier words animal and street did not change either. Yet the meaning of it changed because the surrounding context changed.

There is an important timing detail here.

A human reads the completed sentence and can use the later word tired or flooded to interpret the earlier it. A bidirectional Transformer encoder can also let the representation at it use tokens on both sides.

A decoder-only generative LLM works causally. When it reaches the it position, tired or flooded is still in the future, so it cannot attend to it. When the model later processes tired or flooded, that later position can attend backward to it, animal and street. The model can use the distinction for later predictions, but it does not travel backward in time and rewrite the earlier it representation.

We will keep this difference visible throughout the article:

Attention setup	Context available at it
Bidirectional encoder reading the complete sentence	Earlier and later tokens
Causal decoder processing left to right	Only it and earlier tokens

This is the problem attention helps a language model solve:

When processing one token, which other tokens contain useful information—and how much information should be taken from each one?

Attention is not a database lookup that selects one word and ignores all the others. It usually assigns a weight to every allowed token, then mixes their information according to those weights.

flowchart TD
    Current["Current token: it"] --> Compare["Compare with available tokens"]
    Compare --> Weights["Assign attention weights"]
    Weights --> Mix["Mix their information"]
    Mix --> Contextual["Context-aware representation of it"]

That description is the destination. We will take the slow route to reach it.

We will first understand why embeddings alone are not enough. Then we will follow one token through Query, Key and Value vectors, calculate every dot product, scale the scores, apply softmax and build the final attention output. Only after the numbers make sense will we expand to multi-head attention and real Transformers.

---

Before attention, every token starts with its own vector

A model does not directly operate on words. Text is divided into tokens, each token receives an ID, and that ID retrieves an embedding vector.

For teaching purposes, imagine these small vectors:

animal → [0.9, 0.2]
street → [0.1, 0.8]
it     → [0.5, 0.5]
tired  → [0.8, 0.3]

A vector is an ordered list of numbers. In a real model, each vector may contain hundreds or thousands of values.

The embedding for animal can carry learned information associated with that token. But the initial embedding is not yet specific to this sentence.

Consider the word bank:

I deposited money in the bank.
We sat beside the river bank.

The same token begins with the same learned embedding in both sentences. Its meaning becomes context-specific only as Transformer layers allow it to gather information from surrounding tokens.

Attention is one of the main operations that creates those contextual representations.

---

A useful—but incomplete—library analogy

Imagine entering a library with a question.

You do not combine every book equally. You compare your question with the labels or descriptions of the books, decide which ones appear relevant, and then collect useful information from them.

Attention uses three related ideas:

Attention term	Library analogy	Question it answers
Query	What you are looking for	“What information do I need?”
Key	Label describing what a source offers	“Could this source be relevant?”
Value	Information carried by the source	“What should I take from it?”

The analogy helps separate matching from information transfer:

• Queries are compared with Keys.
• The resulting weights are applied to Values.

This is easy to mix up. We do not usually multiply a Query by a Value to decide relevance. Query–Key similarity decides how strongly the corresponding Value should contribute.

The analogy is incomplete because an LLM does not read semantic book labels written by a human. Query, Key and Value vectors are learned numerical projections. Their useful behavior emerges during training.

---

Where do Query, Key and Value vectors come from?

Suppose a token currently has representation $x$.

The model owns three learned weight matrices:

$$W_Q,\qquad W_K,\qquad W_V$$

It creates three new vectors:

$$q=xW_Q$$ $$k=xW_K$$ $$v=xW_V$$

The same input representation is projected into three different roles.

flowchart LR
    X["Token representation x"] --> Q["Query q = xWQ"]
    X --> K["Key k = xWK"]
    X --> V["Value v = xWV"]

Calculate one Query, Key and Value instead of assuming them

Let the current representation of it be:

$$x_{it}=[0.5,0.5]$$

For this teaching example, use these small projection matrices:

$$W_Q= \begin{bmatrix} 1 & 1\\ 1 & 1 \end{bmatrix} \qquad W_K= \begin{bmatrix} 1 & 0\\ 0 & 1 \end{bmatrix} \qquad W_V= \begin{bmatrix} 1 & 0\\ 0 & 1 \end{bmatrix}$$

Create the Query:

$$q_{it}=x_{it}W_Q$$ $$=[0.5,0.5] \begin{bmatrix} 1 & 1\\ 1 & 1 \end{bmatrix}$$

For the first output position:

$$(0.5\times1)+(0.5\times1)=1$$

The second position has the same calculation, so:

$$\boxed{q_{it}=[1,1]}$$

Now create the Key:

$$k_{it}=x_{it}W_K=[0.5,0.5]$$

And the Value:

$$v_{it}=x_{it}W_V=[0.5,0.5]$$

$W_K$ and $W_V$ happen to be identity matrices here, so they preserve the two input values. $W_Q$ transforms them into [1,1]. Real learned matrices are larger and do not usually look this tidy; these values are chosen so that every multiplication remains visible.

This calculation gives us the exact Query, Key and Value later used for it in our attention walkthrough. The other tokens go through the same three kinds of projection, producing their own vectors.

The matrices are parameters, like the weights from our Forward Pass and Backpropagation articles. During training:

1. attention contributes to a prediction;
2. the model receives a loss;
3. backpropagation calculates gradients for $W_Q$, $W_K$ and $W_V$;
4. an optimizer updates them.

No engineer manually tells one head that animal is a noun or that it is a pronoun. Training gradually shapes the projections because certain patterns help reduce prediction loss.

---

Every token creates all three

It is tempting to think that one token becomes a Query while other tokens become Keys and Values.

In self-attention, every token normally produces its own Query, Key and Value.

For a sequence of four tokens:

Token	Query	Key	Value
animal	$q_1$	$k_1$	$v_1$
street	$q_2$	$k_2$	$v_2$
because	$q_3$	$k_3$	$v_3$
it	$q_4$	$k_4$	$v_4$

The Query belonging to it asks which allowed Keys are relevant to it at this layer. The Query belonging to street asks its own question. Therefore, the attention pattern can be different for every token position.

---

Follow only the word “it” first

Calculating attention for every token at once can hide the intuition. Let us follow one row: the attention produced for it.

Our simplified sequence contains four relevant positions:

animal, street, because, it

This is the context available at the it position in our causal example. We have deliberately stopped before tired; a causal mask would not permit it to use that future token.

From the projection we just calculated, the Query for it is:

$$q_{it}=[1,1]$$

Assume the available Keys are:

Token	Key vector
animal	$[1.5,1.0]$
street	$[0.2,0.8]$
because	$[0.1,0.1]$
it	$[0.5,0.5]$

These are deliberately small teaching values. In a trained model, they would be produced by multiplying token representations by $W_K$.

The first job is to compare the Query of it with each Key.

---

Dot product: a compatibility score

Attention commonly compares a Query and a Key using a dot product.

For two vectors:

$$[a,b]\cdot[c,d]=(a\times c)+(b\times d)$$

Multiply matching positions, then add the results.

Compare it with animal

$$[1,1]\cdot[1.5,1.0]$$ $$=(1\times1.5)+(1\times1.0)=2.5$$

Compare it with street

$$[1,1]\cdot[0.2,0.8]$$ $$=(1\times0.2)+(1\times0.8)=1.0$$

Compare it with because

$$[1,1]\cdot[0.1,0.1]=0.2$$

Compare it with itself

$$[1,1]\cdot[0.5,0.5]=1.0$$

Collect the raw scores:

$$[2.5,1.0,0.2,1.0]$$

The largest score belongs to animal. For this Query and these learned Keys, animal is the most compatible source.

The score 2.5 is not a probability and does not mean “2.5 times relevant.” It is an unnormalized compatibility score.

---

Why use a dot product instead of cosine similarity?

Both operations can compare vectors, but they behave differently.

Cosine similarity divides by vector magnitudes and focuses on angle:

$$\cos(\theta)=\frac{q\cdot k}{\|q\|\|k\|}$$

Scaled dot-product attention uses the dot product directly, followed by scaling and softmax:

$$\operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)$$

This means magnitude can influence the score. The model learns its projections and normalization behavior around that operation. Dot products are also efficient to compute as matrix multiplication on modern hardware.

Attention is therefore not simply “cosine similarity inside an LLM.”

---

Why divide by the square root of the Key dimension?

Our Key vectors contain two numbers, so:

$$d_k=2$$

Scaled attention divides every score by:

$$\sqrt{d_k}=\sqrt{2}\approx1.414$$

The scaled scores become:

Token	Raw score	Scaled score
animal	2.5	1.768
street	1.0	0.707
because	0.2	0.141
it	1.0	0.707

Why is scaling needed?

When vector dimensions are large, dot products can grow large in magnitude. Softmax then becomes extremely sharp: one position may receive almost all the weight, while the others receive values close to zero. In that saturated region, gradients can become unhelpfully small.

Dividing by $\sqrt{d_k}$ keeps score magnitudes in a more manageable range.

The square root is not arbitrary. Under common assumptions where Query and Key components have roughly unit variance, the variance of their dot product grows with $d_k$. Dividing by $\sqrt{d_k}$ brings the scale back under control.

You do not need probability theory to follow the rest. The practical intuition is enough:

More dimensions can create larger dot products, so scale them before softmax.

---

Softmax turns scores into attention weights

Our scaled scores are:

$$[1.768,0.707,0.141,0.707]$$

Softmax exponentiates and normalizes them:

$$\alpha_i=\frac{e^{s_i}}{\sum_j e^{s_j}}$$

The resulting attention weights are approximately:

Source token	Attention weight
animal	52.94%
street	18.33%
because	10.41%
it	18.33%

They are positive and sum to 100%.

Notice that attention did not make a hard decision such as:

animal = selected
everything else = deleted

It created a distribution. animal contributes the most, while other tokens still contribute some information.

This is why “attention means choosing the most important word” is an oversimplification. Attention usually performs a weighted mixture.

---

Scores choose the Values; Values carry the information

Now introduce the Value vectors:

Token	Value vector
animal	$[1.0,0.2]$
street	$[0.1,1.0]$
because	$[0.2,0.1]$
it	$[0.5,0.5]$

Multiply each Value by its attention weight.

From animal:

$$0.5294\times[1.0,0.2]=[0.5294,0.1059]$$

From street:

$$0.1833\times[0.1,1.0]=[0.0183,0.1833]$$

From because:

$$0.1041\times[0.2,0.1]=[0.0208,0.0104]$$

From it:

$$0.1833\times[0.5,0.5]=[0.0916,0.0916]$$

Add them position by position:

$$z_{it} =[0.5294,0.1059] +[0.0183,0.1833] +[0.0208,0.0104] +[0.0916,0.0916]$$ $$\boxed{z_{it}\approx[0.6601,0.3912]}$$

This is the attention output for it in our tiny example.

The original representation of it has now gathered information from the allowed context—especially animal.

flowchart LR
    Scores["Query-Key scores"] --> Softmax["Attention weights"]
    Values["Value vectors"] --> Mix["Weighted sum"]
    Softmax --> Mix
    Mix --> Output["Context-aware output"]

---

The complete calculation in one line

For one Query:

$$\operatorname{Attention}(q,K,V) =\operatorname{softmax}\left(\frac{qK^T}{\sqrt{d_k}}\right)V$$

Read it from left to right:

1. Compare the Query with every Key using $qK^T$.
2. Scale the scores by $\sqrt{d_k}$.
3. Apply softmax to obtain attention weights.
4. Use those weights to combine the Values.

The formula looks compact because matrix operations hide many small dot products and weighted additions.

Our full trace was:

Query for it
    [1, 1]

Dot products with Keys
    [2.5, 1.0, 0.2, 1.0]

Divide by √2
    [1.768, 0.707, 0.141, 0.707]

Softmax
    [0.5294, 0.1833, 0.1041, 0.1833]

Weighted sum of Values
    [0.6601, 0.3912]

---

From one Query to the entire sequence

We calculated only the row belonging to it.

In self-attention, the model creates a Query for every token and compares it with every allowed Key.

Stack all Query vectors into matrix $Q$, all Keys into $K$, and all Values into $V$.

If the sequence has four tokens and each Query/Key has two values:

Q shape = [4, 2]
K shape = [4, 2]
V shape = [4, value_dimension]

Now calculate:

$$QK^T$$

The shapes are:

$$[4,2]\times[2,4]=[4,4]$$

The result is a score matrix:

Query ↓ / Key →	animal	street	because	it
animal	score	score	score	score
street	score	score	score	score
because	score	score	score	score
it	2.5	1.0	0.2	1.0

Each row answers:

For this Query position, how compatible is every Key position?

Softmax is applied across each row, so every Query gets its own attention distribution. Multiplying the attention matrix by $V$ produces one contextual output vector per Query position.

---

Why is it called self-attention?

It is self-attention because Queries, Keys and Values come from the same sequence of representations.

one sequence → Q
same sequence → K
same sequence → V

This allows words in a sentence to exchange information with other words in that sentence.

Cross-attention is different. Queries come from one sequence or component, while Keys and Values come from another.

In the original encoder–decoder Transformer:

decoder states → Queries
encoder output → Keys and Values

In a text-to-image system, text representations may provide context that image representations attend to. The exact direction depends on the architecture.

The matching rule is similar, but the information sources differ.

---

An LLM must not look into the future

During next-token training, suppose the sequence is:

The sky is blue

The position representing is may use earlier context to predict blue. It must not inspect the already-known future token blue, or training would become cheating.

A causal mask blocks future positions.

Before softmax, forbidden scores are replaced with a very large negative value, conceptually $-\infty$.

Why before softmax?

Because:

$$e^{-\infty}=0$$

After softmax, blocked positions receive zero attention weight.

For four positions, the permission pattern looks like:

Query position	May attend to
1	1
2	1, 2
3	1, 2, 3
4	1, 2, 3, 4

flowchart TD
    Scores["Raw attention scores"] --> Mask["Block future positions"]
    Mask --> Softmax["Softmax"]
    Softmax --> Weights["Future weights become zero"]

Models that encode a complete input for classification may use bidirectional attention, where tokens can attend both left and right. Decoder-only generative LLMs normally use causal attention.

---

Padding masks solve a different problem

When sequences of different lengths are placed in one batch, shorter sequences may be padded to a common length.

Sequence A: [real, real, real, real]
Sequence B: [real, real, PAD,  PAD ]

Padding tokens are not meaningful context. A padding mask prevents attention from treating them as ordinary content.

Do not confuse the masks:

Mask	What it blocks
Causal mask	Future content
Padding mask	Artificial padding positions

Some implementations combine their effects before softmax.

---

Position still matters

Self-attention by itself compares vector content. Without positional information, it does not inherently know whether dog appeared before or after bites.

Compare:

dog bites man
man bites dog

The tokens are the same; the order changes the meaning.

Transformers therefore inject or encode position information. Different architectures use learned position embeddings, sinusoidal encodings, rotary position embeddings or other methods.

Attention then works with representations that contain both token-related and position-related information. Position affects which relationships the learned Query and Key projections can express.

Attention does not replace positional information. The two work together.

---

Why have more than one attention head?

One attention calculation creates one way of comparing tokens and mixing information.

But language contains many simultaneous relationships:

• pronoun to possible noun;
• adjective to noun;
• verb to subject;
• closing bracket to opening bracket;
• current token to recent local context;
• question words to relevant facts earlier in the prompt.

Multi-head attention creates several learned attention projections in parallel.

For self-attention head $i$, all three projections begin with the input representation matrix $X$:

$$head_i=\operatorname{Attention} \left( XW_Q^{(i)}, XW_K^{(i)}, XW_V^{(i)} \right)$$

The head outputs are concatenated and projected:

$$\operatorname{MultiHead}(X) =\operatorname{Concat}(head_1,\ldots,head_h)W_O$$

Here, each head owns different $W_Q^{(i)}$, $W_K^{(i)}$ and $W_V^{(i)}$ matrices. We use $X$ rather than already-projected $Q$, $K$ and $V$ so the notation does not accidentally imply that the projections happen twice.

flowchart TD
    X["Input representations"] --> H1["Head 1"]
    X --> H2["Head 2"]
    X --> H3["Head 3"]
    H1 --> Join["Concatenate"]
    H2 --> Join
    H3 --> Join
    Join --> Project["Output projection"]

Different heads can learn different useful patterns because they own different projection matrices. But we should not claim that every head has one clean human-readable job. Some heads may appear specialized, some may combine several behaviors, and some may be redundant.

“One head for grammar, one head for facts” is a helpful cartoon—not a guarantee.

---

A head is usually smaller than the full model width

Suppose the model width is:

$$d_{model}=768$$

and it uses 12 heads. A common arrangement gives each head dimension:

$$d_{head}=768/12=64$$

Each head performs attention in its own 64-dimensional projected space. The 12 outputs are joined back into a 768-dimensional representation.

More heads do not automatically mean more total representation width. The model often divides the available width among them.

Architectures vary, so these numbers are an example rather than a universal rule.

---

Attention is only part of a Transformer block

A Transformer block does more than attention.

A simplified decoder block contains:

flowchart TD
    Input["Input states"] --> Norm1["Normalization"]
    Norm1 --> Attention["Causal self-attention"]
    Attention --> Add1["Residual addition"]
    Input --> Add1
    Add1 --> Norm2["Normalization"]
    Norm2 --> FFN["Feed-forward network"]
    FFN --> Add2["Residual addition"]
    Add1 --> Add2

Exact ordering differs across architectures, but the important roles are:

• attention moves and combines information across token positions;
• the feed-forward network transforms information within each position;
• residual connections preserve a direct information path;
• normalization helps stabilize deep computation.

A model stacks many such blocks. Therefore, the representation of a token can be refined repeatedly.

In an early layer, it may gather local syntactic clues. In later layers, its representation can combine information that earlier tokens have already collected. Context is built across layers, not solved by one magical attention matrix.

---

The same attention example in Python

This code reproduces our calculations without a deep-learning framework:

import math
import numpy as np


x_it = np.array([0.5, 0.5])

W_Q = np.array([
    [1.0, 1.0],
    [1.0, 1.0],
])

W_K = np.array([
    [1.0, 0.0],
    [0.0, 1.0],
])

W_V = np.array([
    [1.0, 0.0],
    [0.0, 1.0],
])

# Project the representation of "it" into three roles.
query_it = x_it @ W_Q
key_it = x_it @ W_K
value_it = x_it @ W_V

keys = np.array([
    [1.5, 1.0],  # animal
    [0.2, 0.8],  # street
    [0.1, 0.1],  # because
    [0.5, 0.5],  # it
])

values = np.array([
    [1.0, 0.2],  # animal
    [0.1, 1.0],  # street
    [0.2, 0.1],  # because
    [0.5, 0.5],  # it
])


def softmax(numbers):
    shifted = numbers - np.max(numbers)
    exponentials = np.exp(shifted)
    return exponentials / exponentials.sum()


# Compare the Query of "it" with every Key.
raw_scores = query_it @ keys.T

# Keep dot products controlled as vector width grows.
scaled_scores = raw_scores / math.sqrt(keys.shape[1])

# Turn scores into positive weights that sum to one.
attention_weights = softmax(scaled_scores)

# Mix the Value vectors using those weights.
attention_output = attention_weights @ values

print("Query for it:", query_it)
print("Key for it:", key_it)
print("Value for it:", value_it)
print("raw scores:", raw_scores)
print("scaled scores:", scaled_scores)
print("attention weights:", attention_weights)
print("weights sum:", attention_weights.sum())
print("attention output:", attention_output)

Expected output is approximately:

Query for it:     [1.0, 1.0]
Key for it:       [0.5, 0.5]
Value for it:     [0.5, 0.5]
raw scores:       [2.5, 1.0, 0.2, 1.0]
scaled scores:    [1.7678, 0.7071, 0.1414, 0.7071]
attention weights:[0.5294, 0.1833, 0.1041, 0.1833]
weights sum:       1.0
attention output: [0.6601, 0.3912]

---

What changes during text generation?

Suppose the prompt contains 1,000 tokens and the model is generating token 1,001.

The new token position needs a Query. It compares that Query with Keys from the allowed earlier positions and combines their Values.

After the model generates a token, that token joins the context. On the next step, the model processes another Query against an even longer history.

Naively recomputing Keys and Values for every earlier token on every generation step would waste work. Earlier tokens have not changed.

This leads to the KV cache.

---

KV cache: remember earlier Keys and Values

During autoregressive generation, the model can store the Keys and Values already calculated for earlier tokens.

Prompt processing:
calculate Keys and Values for prompt tokens → store them

Next generated token:
calculate its new Query, Key and Value
→ compare new Query with cached Keys plus new Key
→ append new Key and Value to cache

Why cache Keys and Values but not all earlier Queries?

At the current decoding step, we need the new position’s Query to look back at the context. Earlier Queries already produced their outputs in earlier steps; we do not need them to compute the new row of causal attention.

KV caching greatly reduces repeated computation during decoding, but it uses memory. The cache grows with factors including:

• number of cached tokens;
• number of layers;
• number and width of stored Key/Value heads;
• batch size;
• numerical precision.

This is one reason long-context inference can be memory-intensive.

Some architectures use multi-query or grouped-query attention to reduce KV cache size by sharing Key/Value heads across multiple Query heads.

---

Prefill and decode feel different to the hardware

LLM inference is often divided into two phases.

Prefill

The model processes the prompt. Many prompt-token operations can be performed in parallel, creating initial hidden states and filling the KV cache.

Decode

The model produces new tokens one at a time. Each step depends on the token selected in the previous step.

Prefill often emphasizes large parallel matrix operations. Decode repeatedly reads the growing KV cache and performs smaller sequential steps. This is why systems separately discuss metrics such as time to first token and time per output token.

Attention is not the only cost in either phase, but it strongly influences long-context serving behavior.

---

Why long sequences are expensive

With ordinary full self-attention, every Query compares with every Key.

For a sequence length $n$, the score matrix contains roughly:

$$n\times n=n^2$$

entries per head.

Sequence length	Pairwise score positions
1,000	1,000,000
2,000	4,000,000
4,000	16,000,000
8,000	64,000,000

Doubling sequence length creates four times as many pairwise positions in the full score matrix.

This quadratic relationship motivates optimized exact-attention implementations, memory-efficient kernels and architectures that restrict or structure which tokens can interact.

However, “attention is $O(n^2)$” needs context. Actual runtime and memory depend on implementation, hardware, head configuration, caching, batch size and whether we are training, prefilling or decoding.

---

FlashAttention changes the implementation, not the definition

A straightforward implementation may write the large attention score matrix to slow high-bandwidth memory and read it again for later operations.

FlashAttention reorganizes the exact calculation into tiles so that more work happens using faster on-chip memory and fewer expensive memory transfers are required. It also avoids materializing the entire attention matrix in the same naive way.

The mathematical result is still attention. FlashAttention is primarily an efficient algorithm for computing it, not a new meaning of Query, Key or Value.

This distinction is important:

attention formula → what is calculated
efficient kernel  → how hardware calculates it

---

Does high attention weight explain the model’s reasoning?

Not reliably by itself.

An attention map can show how one head distributed its weights for one layer and one Query. That can be useful for inspection.

But the model output also depends on:

• many heads;
• many layers;
• Value vectors;
• output projections;
• residual streams;
• feed-forward networks;
• nonlinear interactions.

A high weight does not automatically prove that a token caused the final answer, and a low weight in one head does not prove irrelevance to the complete model.

Attention weights are internal signals, not a guaranteed human-readable chain of thought.

---

Attention is dynamic, not a stored fact table

The model does not store one permanent rule saying:

it always attends 52.94% to animal

The attention weights are recalculated from the current representations and the context that the attention mask permits, for the current input, layer, head and token position.

In a bidirectional encoder, changing tired to flooded can change the representation calculated at the earlier it position because both directions are visible.

In a causal decoder, it cannot change the already-calculated attention row for it. Instead, the later tired or flooded position—and positions generated after it—can attend backward to the earlier context. Their attention patterns and representations can differ. Changing any token inside a Query’s allowed past context can also change that Query’s weights.

This dynamic behavior is what makes attention useful: relevance depends on the currently available context.

---

Common misunderstandings

“Attention means the model understands like a human”

Attention is a learned numerical information-routing operation. It can support impressive language behavior without proving human-like understanding.

“Attention selects exactly one word”

Softmax usually produces a distribution over all allowed positions. Several tokens can contribute at once.

“Query, Key and Value are the original embeddings”

They are learned projections of the representations entering that attention layer.

“Keys contain words and Values contain definitions”

Keys and Values are vectors. The label/information analogy explains their roles, not their literal contents.

“The largest dot product is already a probability”

Dot products are raw compatibility scores. Scaling and softmax produce the attention weights.

“Attention weights are model confidence”

An attention weight describes information mixing inside a head. It is not the same as the final probability assigned to an output token.

“More attention heads always make a model better”

Head count interacts with model width, data, architecture and compute. More is not automatically better.

“The model attends equally to the whole prompt”

Weights vary by layer, head and Query. Context-window availability also does not guarantee that every distant detail will be used effectively.

“KV cache teaches the model during conversation”

The cache stores intermediate Keys and Values for efficient inference. It does not update the model’s trained weights.

“Long context gives free unlimited memory”

Longer context consumes computation and memory, and a model may not use every part equally well. Context is not the same as permanent memory.

---

How engineers debug attention

When an attention implementation behaves incorrectly, useful checks include:

• Are $Q$, $K$ and $V$ shapes correct?
• Was $K$ transposed on the correct dimensions?
• Are scores divided by $\sqrt{d_k}$?
• Is softmax applied across the Key dimension?
• Does each allowed row sum to approximately one?
• Are future positions truly masked in causal attention?
• Are padding positions blocked?
• Are mask values and data types numerically safe?
• Do any scores or weights contain NaN or infinity?
• Are heads reshaped and combined in the correct order?
• During cached decoding, are new Keys and Values appended to the correct layer and position?

A shape trace might look like:

Tensor	Example shape	Meaning
Input	[batch, sequence, model_width]	Contextual token states
Q	[batch, heads, sequence, head_width]	What each position seeks
K	[batch, heads, sequence, head_width]	What each position matches on
V	[batch, heads, sequence, value_width]	Information available to mix
Scores	[batch, heads, query_length, key_length]	Pairwise compatibility
Weights	Same as scores	Normalized attention distribution
Head output	[batch, heads, query_length, value_width]	Mixed Values

Writing the shapes beside each operation catches many errors before inspecting individual numbers.

---

The one idea to remember

Attention answers one practical question:

For this token, at this layer, which available token representations contain useful information, and how should that information be mixed?

It creates Queries, Keys and Values:

$$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V$$

It compares Queries with Keys:

$$QK^T$$

It scales and normalizes the scores:

$$A=\operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)$$

It uses the weights to mix Values:

$$Z=AV$$

Our Query for it gave the largest weight to animal, then produced a new context-aware vector:

$$[0.6601,0.3912]$$

A real Transformer repeats this process across many tokens, heads and layers, with learned vectors far larger than our two-number example.

Attention does not turn text into human thought. It gives the model something more concrete and computationally useful:

a dynamic way to route information through context.

That operation is one of the central reasons Transformers can work with language, code, images and other sequences.

---

Sources and further reading

• Vaswani et al.: Attention Is All You Need
• The Annotated Transformer
• PyTorch: MultiheadAttention
• PyTorch: Scaled Dot Product Attention
• Dao et al.: FlashAttention