Self-Attention

The Attention article explained the general query-key-value mechanism without specifying exactly which tokens get compared against which. This article covers the specific, dominant variant: self-attention.

The simple definition

Self-attention is attention applied within a single sequence — every token calculates its query, key, and value from, and compares itself against, every other token in that same sequence. The “self” specifically means the queries, keys, and values all come from the same source — recall from the Attention article’s general query-key-value mechanism, and note that self-attention is simply that mechanism applied with the input attending to itself, rather than to some separate, different sequence.

Why “self” is the important, distinguishing word here

This is worth being precise about, since the original Transformer architecture, as covered in the Encoder and Decoder articles, actually uses attention in more than one configuration. In self-attention, a sentence looks at itself — every word in “the cat sat on the mat” compares itself against every other word in that same sentence. The original 2017 Transformer’s decoder also uses a different variant, called cross-attention, where the decoder’s queries come from the sequence being generated, but the keys and values come from a separate sequence — the encoder’s output. Self-attention is queries, keys, and values all drawn from the same sequence; cross-attention draws them from two different sequences instead.

flowchart LR
    A[Self-Attention: Query, Key, Value all from the same sequence] --> B["'cat' compares itself against every word in its own sentence"]
    C[Cross-Attention: Query from one sequence, Key/Value from another] --> D["Decoder's query compares against the Encoder's separate output"]

Why self-attention specifically became the dominant choice for decoder-only models

Recall from the Decoder article that most modern large language models, including GPT, are decoder-only — no separate encoder half exists to provide a different sequence for cross-attention to draw from. This is precisely why self-attention, rather than cross-attention, is the mechanism actually doing the heavy lifting in GPT, Gemini, and Claude: the entire ongoing conversation — the user’s messages and the model’s own generated tokens so far — is treated as one single sequence, and every part of that sequence attends to every other part of that same sequence, exactly the “self” in self-attention.

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a group discussion where every participant listens to and weighs the input of every other participant in the same meeting, forming their own updated opinion based on everyone else present — as opposed to a translator in a different meeting entirely, comparing what’s being said in this room against notes from a completely separate meeting held earlier (cross-attention).

Where this breaks down: A group discussion involves genuine back-and-forth conversation over time. Self-attention happens as a single, simultaneous mathematical calculation — every token’s query compared against every other token’s key, all at once, through the parallel processing described in the Transformer article, not a sequential back-and-forth exchange the way a real discussion unfolds.

What self-attention actually reveals about a sentence

This is worth grounding concretely, since self-attention’s real value shows up in exactly the kind of within-sentence relationships a single piece of text contains. In “The animal didn’t cross the street because it was too tired,” self-attention lets “it” strongly attend to “animal” (not “street”), correctly resolving the pronoun using information found entirely within that same sentence — no separate, external sequence needed, since everything required to resolve the ambiguity is present in the sentence itself. This is precisely the kind of relationship self-attention was built to capture, and it’s the same underlying mechanism referenced in the Attention article’s trophy-and-suitcase example.

A concrete example, layered

For a simple beginner example: self-attention processing “She poured water from the pitcher into the cup until it was full” lets the model calculate that “it” refers to “cup” (the thing being filled), not “pitcher” (the thing being poured from) — resolved entirely through comparing every word in the sentence against every other word in that same sentence. For a production example: GPT-3’s decoder-only architecture, as confirmed in OpenAI’s published paper, relies entirely on masked self-attention (the backward-looking variant covered in the Decoder article) — there’s no separate encoder, no cross-attention, just self-attention applied repeatedly across 96 layers to the single, growing sequence of the conversation so far.

Self-attention versus cross-attention

Mechanism	Queries come from	Keys and values come from	Example
Self-attention	Current sequence	The same sequence	Words relate to other words in a prompt.
Cross-attention	Decoder sequence	Encoder output	Translation decoder reads the encoded source sentence.

The word self means Q, K, and V are created from the same sequence representation. It does not mean the model is conscious or attending to itself psychologically.

Encoder and decoder self-attention differ

flowchart TB
    A[Encoder self-attention] --> B[Every input position can usually see both directions]
    C[Decoder self-attention] --> D[Causal mask hides future positions]

GPT uses causal self-attention so generation cannot look ahead. Encoder models use bidirectional self-attention when the full input is already known. Gemini’s decoder foundation likewise supports generative causal processing while incorporating multimodal context.

Why attention cost grows quickly

With sequence length n, ordinary self-attention forms roughly an n × n score table for each head:

128 tokens   → 16,384 token-pair scores
1,024 tokens → 1,048,576 token-pair scores

This quadratic growth is one reason long-context attention requires memory- and compute-efficient techniques.

Trace one token through self-attention

Take “The bank beside the river flooded.” The raw word bank could mean a financial company or land beside water.

The current vector for bank is projected into its query, key, and value vectors.
Its query is compared with keys from visible tokens such as beside, river, and flooded.
The scores are scaled and passed through softmax.
The weighted value vectors are combined.
The updated vector for bank now carries stronger evidence for the river-bank meaning.

flowchart LR
    B[bank: initially ambiguous] --> A[Self-attention]
    R[river] --> A
    F[flooded] --> A
    A --> U[bank: context-aware representation]

This does not permanently change the embedding stored for the word. It creates a contextual representation for this occurrence of bank in this particular sequence.

Self-attention in text, code, images, and video

In text, it can connect a pronoun with the noun it refers to.
In code, it can connect a function call with relevant definitions or variables.
In an image represented as patches, it can relate one image region to another.
In video, it can connect information across encoded frames.

Google’s current Gemini long-context guide describes models processing text, video, audio, and images in large contexts. The media must first become token-like numeric representations; attention then operates on those representations rather than directly “looking” or “listening” like a person.

Common misconception

A frequent beginner assumption: that self-attention and attention are simply two different words for the exact same thing. As this article has explained, they’re related but distinct — attention is the general mechanism, and self-attention is the specific case where a sequence attends to itself, as opposed to cross-attention, where one sequence attends to a separate, different sequence. Knowing this distinction matters especially for understanding encoder-decoder architectures, where both self-attention and cross-attention genuinely coexist and serve different roles.

Where this fits in what comes next

You now understand the specific attention variant powering most modern language models. The next article, Multi-Head Attention, covers a detail already glimpsed in the Attention article in full — how Transformers run several of these self-attention calculations in parallel, each specializing in a different kind of relationship — before Positional Encoding addresses a real gap this article’s mechanism leaves open: since self-attention compares every token against every other token simultaneously, with no inherent sense of order, how does a Transformer know that “dog bites man” and “man bites dog” are different sentences at all?

In one sentence

Self-attention is attention applied within a single sequence — every token comparing itself against every other token in that same sequence — and it’s the specific mechanism, distinct from cross-attention, that decoder-only architectures like GPT rely on entirely, with no separate encoder needed at all.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026