Embeddings and Representations

Before you continue: three tools for this module

Vector: an ordered list of numbers.
Embedding: a learned vector representation.
Similarity: a numerical comparison between representations.

You do not need to memorize these yet. Use this map when the terms reappear.

Begin with the central question

What hidden problem does Embeddings and Representations solve inside a real language-model system?

Keep that central question about Embeddings and Representations in mind. The definitions, numbers, diagrams, and code examples below answer it one piece at a time.

token or text → learned vector → contextual transformation or similarity

1. What You Will Learn

Learning outcomes

Distinguish token embeddings, contextual hidden states, and output logits.
Explain how text becomes vectors that a neural network can transform.
Use similarity carefully and recognize incompatible embedding spaces.
Connect representations to semantic search, RAG, and internal LLM processing.

In one sentence

💡 Big picture

Embeddings turn tokens or text into lists of numbers, giving the model a mathematical way to represent and compare information.

2. Why This Module Exists

The problem this module solves

Text must become numbers before neural-network layers can process it.
You need to distinguish the starting embedding from the changing hidden representation created inside later layers.

3. Intuition

a token’s journey through an LLM isn’t a single lookup — it’s a starting point (the embedding) that gets progressively reshaped, layer by layer, as it incorporates more and more context from surrounding tokens. By the final layer, the vector representing “bank” in a sentence about loans looks measurably different from where it started, precisely because it’s absorbed information from “approved” and “loan” along the way.

Analogy: The Traveler’s Passport & Contextual Stamps Think of a token’s journey through successive model layers in terms of travel stamps:

The Token Embedding (The Blank Passport): You lookup the word “bank”. It gets the exact same starting vector (a blank passport) regardless of the sentence it resides in. No destination is printed on it.

The Hidden State (The Journey):

At layer 1, you visit the city “approved”. You get a stamp.

At layer 2, you visit the city “loan”. You get another stamp.

At each floor, self-attention modifies your vector with contextual offsets, pulling the representation closer to financial concepts.

The Final Representation (The Stamped Passport): By the final layer, your passport is packed with stamps. If we measure its cosine similarity to the starting blank passport, it has drifted down to 0.4292 similarity — it is now a highly specific, context-stamped passport.

📊 Visual Chart: Cosine Similarity Drift Across Layers

Here is how the contextual vector moves away from the static, context-free token embedding:

graph TD
    classDef lookup fill:#e74c3c,stroke:#333,stroke-width:1px,color:#fff;
    classDef drift fill:#3498db,stroke:#333,stroke-width:1px,color:#fff;

    Lookup["Token Embedding Lookup (Layer 0)<br>(Cosine Similarity = 1.0000)"]:::lookup --> Layer1["Layer 1 Hidden State<br>(Cosine Similarity = 0.4934)"]:::drift
    Layer1 --> Layer2["Layer 2 Hidden State<br>(Cosine Similarity = 0.4495)"]:::drift
    Layer2 --> Layer3["Layer 3 Hidden State<br>(Cosine Similarity = 0.2829)"]:::drift
    Layer3 --> Layer4["Layer 4 Final Representation<br>(Cosine Similarity = 0.4292)"]:::drift

    Layer4 --> LMHead["Output Projection Head (predict next token)"]

4. Core Concept — Three Distinct Terms

Token embedding:      the FIXED, LEARNED vector looked up once
                       per token ID, from the embedding table --
                       identical every time this token ID appears,
                       with NO context incorporated yet (NLP
                       course Module 9's proven limitation)

Hidden state:            the representation AT ANY GIVEN LAYER,
                       during processing -- recomputed layer by
                       layer, incorporating progressively more
                       context via attention

Final representation:      the hidden state AFTER THE LAST
                       Transformer layer -- what actually gets
                       passed to the LM head (Module 5) to
                       produce logits

Token ID
   ↓
Embedding lookup           (token embedding — fixed, no context)
   ↓
+ Positional information
   ↓
Transformer layer 1          (hidden state #1)
   ↓
Transformer layer 2            (hidden state #2 — MORE context
                                incorporated)
   ↓
...
   ↓
Transformer layer N              (final representation)
   ↓
LM Head → Logits (Module 5)

5. How Representations Change Through Transformer Layers

You already built and verified this exact mechanism completely in the Transformers course (multi-head attention, residuals, LayerNorm, the full block, stacked). The specific LLM-relevant question this module answers: does a token’s representation actually change meaningfully as it passes through more layers, or does it stay close to its starting embedding? Verified directly below.

6. Mathematical Intuition

Read the mathematics as a story

token or text → learned vector → contextual transformation or similarity

First locate the input, operation, and output. Then treat the formula as a compact description of that journey rather than a collection of symbols to memorize.

Cosine similarity (already used throughout the NLP and Transformers courses) between a token’s representation at layer N and its original starting embedding measures how much that representation has “moved” due to context incorporation. A similarity near 1.0 means little change; a similarity meaningfully below 1.0 means the representation has genuinely been reshaped by attending to surrounding tokens.

7. Small Worked Example

Walk through the example

Name what each input represents.
Follow one transformation at a time.
Translate the result back into ordinary language.

The purpose is to reveal the mechanism, not merely display an answer.

The word “bank” in “the bank approved my loan” starts with a fixed token embedding — identical to whatever “bank” would have in any other sentence (NLP course Module 9’s proven limitation). As it passes through successive Transformer layers, attention lets it incorporate information from “approved” and “loan” — you’d expect its representation to measurably drift away from that original, context-free starting point.

8. Python Example

What the code will demonstrate

The code builds a tiny version of the mechanism, prints values you can inspect, and connects them to the worked example. Predict the direction of the result before running it.

Python symbols used below

NumPy (np) stores numeric vectors and matrices.
np.array(...) creates a numeric collection.
Library calls perform the same conceptual steps shown above at a larger scale.

# Build a small, inspectable example of Embeddings and Representations.
# Follow the inputs, transformations, and output in order.
import numpy as np

def softmax(x, axis=-1):
    exp_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return exp_x / np.sum(exp_x, axis=axis, keepdims=True)

def layer_norm(x, eps=1e-8):
    mean = x.mean(axis=-1, keepdims=True)
    std = x.std(axis=-1, keepdims=True)
    return (x - mean) / (std + eps)

def relu(x): return np.maximum(0, x)

np.random.seed(3)
d_model = 4
seq_len = 5

# Step 1: Token ID -> Embedding lookup (fixed, learned table)
vocab = ["the", "bank", "approved", "my", "loan"]
embedding_table = np.round(np.random.randn(len(vocab), d_model) * 0.4, 3)
token_ids = [0, 1, 2, 3, 4]
token_embeddings = embedding_table[token_ids]

def positional_encoding(n, d):
    pos = np.arange(n)[:, np.newaxis]
    i = np.arange(d)[np.newaxis, :]
    angles = pos / np.power(10000, (2 * (i // 2)) / np.float32(d))
    pe = np.zeros((n, d))
    pe[:, 0::2] = np.sin(angles[:, 0::2])
    pe[:, 1::2] = np.cos(angles[:, 1::2])
    return pe

x = token_embeddings + positional_encoding(seq_len, d_model)

def transformer_block(x, seed):
    rng = np.random.RandomState(seed)
    Wq, Wk, Wv, Wo = [rng.randn(d_model, d_model) * 0.3 for _ in range(4)]
    Q, K, V = x @ Wq, x @ Wk, x @ Wv
    scores = Q @ K.T / np.sqrt(d_model)
    attn_out = (softmax(scores, axis=-1) @ V) @ Wo
    x = layer_norm(x + attn_out)
    W1 = rng.randn(d_model, d_model * 2) * 0.3
    W2 = rng.randn(d_model * 2, d_model) * 0.3
    ffn_out = relu(x @ W1) @ W2
    return layer_norm(x + ffn_out)

def cosine_sim(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

bank_idx = vocab.index("bank")
initial_bank_vector = x[bank_idx].copy()

print("Tracking 'bank' token's representation across layers:")
print(f"  Layer 0 (embedding+pos, no context yet): {np.round(initial_bank_vector, 4)}")

num_layers = 4
for layer_idx in range(num_layers):
    x = transformer_block(x, seed=layer_idx)
    sim_to_original = cosine_sim(x[bank_idx], initial_bank_vector)
    print(f"  After layer {layer_idx+1}: {np.round(x[bank_idx], 4)}  (similarity to original: {sim_to_original:.4f})")

Expected Output:

Tracking 'bank' token's representation across layers:
  Layer 0 (embedding+pos, no context yet): [ 0.7305  0.3983 -0.023   0.749 ]
  After layer 1: [ 1.2804  0.1417 -1.5263  0.1042]  (similarity to original: 0.4934)
  After layer 2: [ 1.4258  0.114  -1.3898 -0.1501]  (similarity to original: 0.4495)
  After layer 3: [ 1.5882  0.1187 -0.9125 -0.7944]  (similarity to original: 0.2829)
  After layer 4: [ 1.5882 -0.1642 -1.2081 -0.2004]  (similarity to original: 0.4292)

9. How It Works

Layer 0 is the token embedding (plus positional info) — fixed, computed once, no surrounding context incorporated.
By layer 1, similarity to the original has already dropped to 0.4934 — attention has pulled in information from “approved,” “my,” and “loan,” genuinely reshaping the representation in just one layer.
Similarity continues shifting through layers 2-4 (0.4495, 0.2829, 0.4292) — never converging back to 1.0, confirming the representation has been durably transformed by context, not merely perturbed and restored.
This is the concrete, numeric meaning of “hidden state” vs. “token embedding”: every intermediate x[bank_idx] after each layer is a hidden state — a progressively more contextual representation — while only layer 0 is the raw token embedding. The final representation (after layer 4 here) is specifically what would feed the LM head (Module 5) to produce next-token logits.

10. How Is This Used in Modern AI?

Trace it through a real model call

user message → assembled context → LLM computation → decoded output → application checks

This topic affects one stage of that path; it is not the complete product. Hosted GPT- and Gemini-style applications also add instructions, safety systems, retrieval, tools, serving infrastructure, and evaluation around the model.

🤖 How Is This Used in Modern AI?

This layer-by-layer contextualization is precisely why an LLM can distinguish “bank” (financial) from “bank” (riverbank) — exactly the NLP course’s proven contextual-representation result, now traced with real numbers showing the representation actively drifting, layer by layer, as more surrounding context is incorporated.

Term	Where it matters practically
Token embedding	The embedding matrix’s parameter count (Module 12); what a model “starts with” before any reasoning
Hidden state	What some embedding APIs let you extract from intermediate layers, for specific use cases
Final representation	What feeds the LM head — directly determines the next-token probability distribution (Module 5)

11. How Is This Used in Agentic AI?

Separate the model from the runtime

goal + state + tool results → LLM proposal → runtime validation → execution or response

The LLM proposes text or a structured action. Ordinary application code controls permissions, tools, retries, memory, and execution.

Direct relevance to Agentic AI: High, foundational. Every piece of an agent’s context — user messages, tool results, retrieved documents — undergoes exactly this layer-by-layer transformation before the model can reason about it.

This is the concrete mechanism behind an agent’s ability to correctly relate a new instruction back to something mentioned many turns earlier: attention, at every layer, actively incorporating that distant context into the current representation.

12. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: using “embedding” and “hidden state” interchangeably.

Why it is incorrect: As demonstrated directly, they’re measurably different — a token embedding is fixed and context-free; a hidden state at any given layer has already incorporated real context, verified by its dropping similarity to the original embedding.

⚠️ Mistake

Incorrect idea: assuming more layers always means “more different” from the original embedding, monotonically.

Why it is incorrect: As shown, similarity doesn’t decrease perfectly monotonically (layer 4’s 0.4292 is slightly higher than layer 3’s 0.2829) — representations evolve in complex, non-linear ways across layers, not a simple, steadily increasing drift.

⚠️ Mistake

Incorrect idea: thinking “final representation” and “logits” are the same thing.

Why it is incorrect: The final representation is a hidden-state-shaped vector; logits (Module 5) are a different, vocabulary-sized vector produced by projecting the final representation through the LM head.

13. Important Distinctions

Token Embedding	Hidden State
Fixed, looked up once per token ID	Recomputed at every layer
No context incorporated	Progressively incorporates context via attention

Hidden State (intermediate layer)	Final Representation
Any layer’s output during processing	Specifically the LAST layer’s output
Not directly used for next-token prediction	Directly fed to the LM head to produce logits (Module 5)

14. When to Use

Not applicable in the technique-choice sense — this module’s value is precise vocabulary and mechanism understanding, directly useful whenever discussing what a model “represents” at a given stage.

15. When Not to Use

Avoid loosely using “embedding” to describe every stage of processing — being precise about token embedding vs. hidden state vs. final representation avoids genuine confusion in technical discussions and documentation.

16. Production Considerations

Some embedding APIs expose intermediate hidden states (not just final representations) for specific use cases — knowing precisely which layer’s output you’re getting matters for interpreting results correctly.
The embedding matrix and LM head are sometimes “tied” (share the same weights) in some model architectures — a practical parameter-efficiency detail worth knowing exists.

17. What You Should Remember

Token embedding = fixed, looked up once, no context. Hidden state = recomputed per layer, progressively contextual. Final representation = the last layer’s hidden state, feeding the LM head.
Verified directly: a token’s representation genuinely, measurably drifts from its starting embedding as it passes through more layers — not a superficial relabeling, a real transformation.
This drift is the mechanism behind contextual understanding — the same underlying proof from the NLP course’s contextual representations module, now traced numerically layer by layer.

18. Interview Questions

Beginner

Q: What’s the difference between a token embedding and a hidden state?

Ans: A token embedding is the fixed vector looked up once for a given token ID, with no surrounding context incorporated — it’s identical every time that token appears anywhere.

A hidden state is the representation at any given layer during processing, which has already incorporated context from surrounding tokens via attention, and generally differs from layer to layer.

Intermediate

Q: What is the “final representation” in an LLM, and what is it used for?

Ans: The final representation is the hidden state produced by the LAST Transformer layer — it’s what gets passed to the LM head (Module 5), which projects it into a vocabulary-sized vector of logits, ultimately producing the next-token probability distribution.

It’s distinct from earlier layers’ hidden states, which contributed to building it but aren’t themselves used for the final prediction.

Advanced

Q: Using this module’s verified result, explain why a token’s representation doesn’t simply stay close to its original embedding as it passes through more Transformer layers.

Ans: Each Transformer layer’s attention mechanism lets a token’s representation incorporate information from other tokens in the sequence, weighted by relevance — and each layer’s feed-forward network further transforms that representation.

This was verified directly: a token’s cosine similarity to its own starting embedding dropped substantially after just one layer (from 1.0 to roughly 0.49) and never returned close to 1.0 through subsequent layers, confirming the representation is genuinely, durably reshaped by context — not merely perturbed and restored to its original form.

Scenario

**Q: A team is building a tool that extracts “embeddings” from a specific layer of an open-source LLM for a downstream similarity task, but results seem inconsistent with what they expected from a standard embedding model.

What might explain this, based on this module?** A: They may be conflating a raw token embedding (Module 8, no context) with a hidden state from an intermediate Transformer layer (this module) — these are genuinely different representations, verified directly to diverge substantially even after just one layer.

If they intended to extract a genuinely contextual, semantically meaningful representation (like what a dedicated embedding model produces), they likely want a later layer’s hidden state, or the final representation specifically — not the raw, context-free token embedding from the very first layer.

AI Engineering

Q: Why does understanding the token embedding / hidden state / final representation distinction matter practically when working with LLM internals or embedding APIs?

Ans: Different tools and APIs may expose different stages of this pipeline — some embedding models return final-layer representations (most semantically rich, most commonly what you want for similarity/ retrieval tasks), while inspecting or fine-tuning tools might expose intermediate hidden states or raw token embeddings for specific diagnostic or research purposes.

Verified directly in this module, these stages produce measurably different vectors — using the wrong one for a given task (e.g., raw token embeddings for a semantic similarity task that needs full contextual understanding) would produce meaningfully worse results than intended.

19. Next Step

Next: Module 5 — Next Token Prediction — one of the most important modules in this course: precisely what “predict the next token” means, mechanically, from logits through softmax to selection.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026