TechByteByByte

Contextual Representations

Understand contextual representations through ELMo and BERT, how surrounding text changes a token representation, and why context sensitivity improves—but does not guarantee—word-sense understanding.

#NLP#AI#Contextual Embeddings#BERT#ELMo

Begin with the central question

Can the same word receive a different vector when its sentence changes?

Essential words

A contextual representation uses surrounding tokens. ELMo uses bidirectional LSTMs. BERT uses bidirectional Transformer self-attention and masked-language-model training.

What You Will Understand

An important improvement in this course’s central thread: Module 9 proved static embeddings assign “bank” one fixed vector, no matter the context. This module demonstrates, directly and numerically, that a context-processing network can address this structural limitation — the same input embedding for “bank” now produces different final representations depending entirely on the surrounding sentence.

token + surrounding tokens -> contextual representation

Why Representations Must Adapt to Context

Module 9’s proof was the central diagnostic of this course. Modules 10-12 built the machinery (RNNs’ hidden states, sequence-to-sequence, attention) piece by piece. This module exists to close the loop explicitly: can surrounding context change the representation that a static lookup could not? The small demonstration below tests that narrower claim.

A Fresh Meaning for Every Sentence

instead of looking up a word’s meaning in a fixed dictionary (static embeddings), a contextual model computes a word’s representation fresh, every time, by looking at its specific neighbors in that specific sentence — using a context-processing network. BERT uses self-attention, while the earlier ELMo model used bidirectional LSTMs rather than attention.

Analogy: The Context-Aware Dictionary Imagine reading a book with two different types of dictionaries:

  • The Static Dictionary (Word2Vec): A heavy, printed book. When you look up “bank”, page 104 permanently reads: “bank (noun): A place that holds money OR land next to a river.” (The Platypus compromise average). You get the exact same printed definition regardless of the sentence you are reading.
  • The Context-Aware Dictionary (BERT/ELMo): An AI-powered e-reader. As you read “I fished from the muddy bank”, the screen highlights “bank” and dynamically generates a single, custom definition: “bank: the muddy ground bordering a body of water.” If you swipe to the next page and read “The bank approved my home loan”, the screen instantly updates the definition to: “bank: a financial institution authorized to receive deposits and issue loans.”
  • The definition is computed fresh every time, determined entirely by reading the surrounding neighbor words.

📊 Visual Flowchart: Bidirectional Context Processing (BERT)

Here is how self-attention layers process surrounding context in both directions to compute context-specific vectors:

graph TD
    subgraph Sentence1 ["Sentence 1: 'the bank approved my loan'"]
        w1_1["the"] <--> w1_2["bank"]
        w1_2 <--> w1_3["approved"]
        w1_3 <--> w1_4["my"]
        w1_4 <--> w1_5["loan"]

w1_2 --> OutputVector1["Output Vector for 'bank':<br>[Financial Institution Context]"]
    end

subgraph Sentence2 ["Sentence 2: 'the river bank was muddy'"]
        w2_1["the"] <--> w2_2["river"]
        w2_2 <--> w2_3["bank"]
        w2_3 <--> w2_4["was"]
        w2_4 <--> w2_5["muddy"]

w2_3 --> OutputVector2["Output Vector for 'bank':<br>[Geological Riverbank Context]"]
    end

4. Core Concept

One-hot                (Module 2 — no meaning at all)
   ↓
TF-IDF                    (Module 5 — weighted counts, still no meaning)
   ↓
Static Word Embeddings       (Module 8 — meaning exists, but ONE
                              vector per word, proven insufficient
                              in Module 9)
   ↓
Contextual Representations       (this module — a DIFFERENT vector
                                  per word, depending on context)
TermDefinition
ELMoAn early contextual representation approach, using bidirectional LSTMs to produce context-dependent word representations
BERTA Transformer-based, encoder-only model (Transformers course Module 12-13) producing deeply contextual representations via bidirectional self-attention
Masked language modelingBERT’s training approach: hide random words, train the model to predict them using surrounding (bidirectional) context
Bidirectional contextUsing BOTH preceding AND following words to build a representation — not just preceding words as in Module 10’s RNN

5. How It Works — Step by Step (conceptual)

1. Instead of a fixed embedding LOOKUP TABLE (Module 8), a
   contextual model computes each word's representation using
   an attention-based mechanism (Module 12) over its ACTUAL
   surrounding words in THIS specific sentence
2. For BERT specifically: bidirectional self-attention means
   a word's representation incorporates BOTH earlier AND later
   words in the sentence -- not just what came before (unlike
   Module 10's RNN, which only sees preceding words)
3. BERT is trained via MASKED LANGUAGE MODELING: randomly hide
   words in training sentences, train the model to predict them
   using the SURROUNDING (both-directions) context -- this
   objective specifically forces the model to build genuinely
   context-dependent representations
4. The RESULT: the same word, in different sentences, produces
   DIFFERENT final representations, verified directly below

6. Mathematical Intuition

The limited test in this module is precise about one thing: start from the same static embedding for “bank” in two different sentences. Run each sentence through an attention mechanism (Module 12). If the resulting, attention-processed representations for \

7. Simple Example

“I deposited money at the bank” and “we sat by the river bank” both start with the identical static embedding for “bank.” After passing through an attention mechanism that lets “bank” draw information from its actual neighboring words in each specific sentence, the two resulting representations should diverge — reflecting each sentence’s distinct context.

8. Build It in Python

What the code will demonstrate

Both sentences begin with exactly the same static vector for “bank.” Self-attention then mixes information from the surrounding tokens, producing a different output vector at the bank position in each sentence.

Different outputs prove that context affected the computation. They do not prove that this untrained toy attention layer correctly understands finance or rivers; semantic usefulness requires training and evaluation.

import numpy as np

def softmax(x, axis=-1):
    exp_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return exp_x / np.sum(exp_x, axis=axis, keepdims=True)

# These repeatable random projections demonstrate context dependence, not learned meaning.
np.random.seed(8)
d = 4

# Static embedding for "bank" -- IDENTICAL regardless of sentence (Module 9)
static_bank_embedding = np.array([0.5, 0.5, 0.3, 0.3])

sentence_a_words = ["i", "deposited", "money", "at", "the", "bank"]
sentence_b_words = ["we", "sat", "by", "the", "river", "bank"]

embeddings_a = {
    "i": np.array([0.2, 0.1, 0.1, 0.1]), "deposited": np.array([0.8, 0.2, 0.1, 0.1]),
    "money": np.array([0.85, 0.15, 0.1, 0.05]), "at": np.array([0.1, 0.1, 0.1, 0.1]),
    "the": np.array([0.05, 0.05, 0.05, 0.05]), "bank": static_bank_embedding,
}
embeddings_b = {
    "we": np.array([0.2, 0.15, 0.1, 0.1]), "sat": np.array([0.15, 0.6, 0.1, 0.1]),
    "by": np.array([0.1, 0.1, 0.1, 0.1]), "the": np.array([0.05, 0.05, 0.05, 0.05]),
    "river": np.array([0.1, 0.85, 0.15, 0.1]), "bank": static_bank_embedding,
}

# Project each token into Query, Key, and Value spaces used by self-attention.
Wq = np.round(np.random.randn(d, d) * 0.4, 2)
Wk = np.round(np.random.randn(d, d) * 0.4, 2)
Wv = np.round(np.random.randn(d, d) * 0.4, 2)

# Build one contextual output vector for every token position in a sentence.
def contextualize(words, embed_dict):
    X = np.array([embed_dict[w] for w in words])
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    scores = Q @ K.T / np.sqrt(d)
    weights = softmax(scores, axis=-1)
    return weights @ V   # CONTEXTUAL representation for every position

contextual_a = contextualize(sentence_a_words, embeddings_a)
contextual_b = contextualize(sentence_b_words, embeddings_b)

# Compare only the output position occupied by the same word “bank.”
bank_idx_a = sentence_a_words.index("bank")
bank_idx_b = sentence_b_words.index("bank")

print("Static 'bank' embedding (SAME in both sentences):", static_bank_embedding)
print("\nCONTEXTUAL representation in sentence A (financial):", np.round(contextual_a[bank_idx_a], 4))
print("CONTEXTUAL representation in sentence B (river):     ", np.round(contextual_b[bank_idx_b], 4))

print("\nAre the two CONTEXTUAL 'bank' representations different?",
      not np.allclose(contextual_a[bank_idx_a], contextual_b[bank_idx_b]))

Expected Output:

Static 'bank' embedding (SAME in both sentences): [0.5 0.5 0.3 0.3]

CONTEXTUAL representation in sentence A (financial): [-0.0144 -0.0261  0.0374 -0.0672]
CONTEXTUAL representation in sentence B (river):      [ 0.0145 -0.1251  0.0137 -0.0438]

Are the two CONTEXTUAL 'bank' representations different? True

9. How It Works

This demonstrates the structural improvement over Module 9’s static lookup, but it is not proof of perfect language understanding. Both sentences start with the identical static embedding for “bank” ([0.5, 0.5, 0.3, 0.3]), exactly as Module 9 demonstrated static embeddings always do. But after passing through an attention mechanism that lets “bank” draw information from its actual surrounding words in each specific sentence, the resulting contextual representations are genuinely different (True).

This is the complete arc this course has been building: Module 9 proved the problem exists → Module 10’s RNN showed hidden states could help → Module 11 proved RNN-based compression has its own bottleneck → Module 12 introduced attention as the fix → this module verifies attention genuinely delivers context-dependent representations, closing the loop entirely.

10. How Is This Used in Modern AI?

🤖 How Is This Used in Modern AI?

Most current neural embedding models used for RAG and semantic search are context-sensitive rather than simple static word lookups — built on exactly this principle, using Transformer architectures (covered fully in the dedicated Transformers course). BERT-style bidirectional encoder models remain genuinely useful for understanding-focused tasks (classification, extraction); decoder-only models (also attention-based, covered fully in the Transformers course) dominate generation-focused tasks.

Model typeBest suited for
BERT-style (encoder, bidirectional)Classification, extraction, understanding-focused tasks
GPT-style (decoder-only, covered in Transformers course)Text generation

Real systems you can recognize

The Hugging Face BERT documentation exposes BERT tokenizers and models for tasks such as masked language modeling, classification, and token classification. BERT computes context-sensitive token states using bidirectional self-attention.

OpenAI embedding models and many Hugging Face sentence-transformer models produce one vector for a complete input passage. That passage vector is context-sensitive, but it is not the same object as one individual BERT token vector.

11. How Is This Used in Agentic AI?

Direct relevance to Agentic AI: Very High. Many production RAG systems use context-sensitive embedding models, precisely so it can correctly distinguish a query about “bank accounts” from one about “riverbanks” — the exact failure Module 9 proved static embeddings suffer from, and the exact failure this module verified contextual representations resolve.

12. Common Mistakes / Misunderstandings

⚠️ Mistake: assuming BERT and GPT-style models are the same kind of architecture. BERT is encoder-based with bidirectional context (sees both earlier and later words); GPT-style models are decoder-only with causal (forward-only) context — a genuine architectural difference, covered fully in the Transformers course.

⚠️ Mistake: thinking this module reintroduces attention mechanics from scratch. It doesn’t — Module 12 covered the NLP motivation; this module’s specific job was verifying the outcome: does context genuinely change a word’s representation? Confirmed directly.

⚠️ Mistake: believing contextual representations eliminate ALL ambiguity challenges perfectly. They dramatically improve on static embeddings, verified directly — but genuinely difficult, highly ambiguous cases can still challenge even contextual models; this isn’t a claim of perfect disambiguation in every case.

13. Important Distinctions

Static Embeddings (Module 8-9)Contextual Representations (this module)
ONE vector per word, alwaysA DIFFERENT vector per word, per context — verified directly
Proven: identical for “bank” regardless of sentenceProven: genuinely different for “bank” across sentences
ELMoBERT
Bidirectional LSTMs (Module 10-style, extended)Bidirectional Transformer self-attention (Module 12-style)
An earlier historical approachThe more modern, attention-based standard
BERT (encoder)GPT-style (decoder-only)
Bidirectional contextCausal (forward-only) context
Best for understanding tasksBest for generation tasks

14. When to Use

Use contextual representations (the standard today) whenever word-sense disambiguation or nuanced semantic understanding genuinely matters — which is most real NLP tasks, given Module 9’s proven limitation of the static alternative.

15. When Not to Use

Static embeddings might still suffice for genuinely simple tasks with low ambiguity risk and tight compute/latency constraints — but for anything requiring real semantic precision, contextual representations are the well-justified standard.

16. Production Considerations

  • Contextual representations require recomputation per sentence/ context — unlike static embeddings, which can be precomputed once and reused, a word’s contextual representation must be recomputed whenever its surrounding context is available, a genuine, real compute cost trade-off (fully covered in the Transformers course’s efficiency module).
  • Modern production embedding models for RAG are commonly context-sensitive — this is a well-established, empirically justified default, not a debatable choice.

17. Interview Questions

Beginner

Q: What’s the key difference between a static and a contextual word representation?

Ans: A static representation assigns exactly one fixed vector to a word, regardless of context. A contextual representation computes a potentially different vector for the same word depending on its surrounding sentence — directly solving the ambiguity problem static embeddings can’t handle.

Intermediate

Q: How does BERT’s bidirectional context differ from the RNN-based context covered in Module 10?

Ans: Module 10’s RNN processes a sequence in one direction (left to right), so its hidden state at any point only reflects PRECEDING words. BERT uses bidirectional self-attention, meaning a word’s representation incorporates BOTH earlier AND later words in the sentence simultaneously — a more complete view of context than a one-directional RNN can provide.

Advanced

Q: Explain, using this module’s verified result, why contextual representations genuinely solve the limitation proven in Module 9.

Ans: Module 9 proved that two sentences using “bank” in different senses (financial vs. riverbank) receive the exact same static embedding for that word, since static embeddings are looked up once and never change based on context. This module started from that same identical static embedding in two different sentences, then passed each sentence through an attention mechanism that lets “bank” draw information from its actual neighboring words.

The resulting vectors are different between the two sentences, demonstrating that the surrounding tokens influenced the computation. This toy result proves context sensitivity, not correct word-sense understanding: a trained model must still be evaluated to determine whether the changed vector captures the useful meaning.

Scenario

Q: A team switches their RAG system’s embedding model from a static Word2Vec-based approach to a modern contextual (Transformer-based) embedding model. Explain, referencing this module and Module 9, what concrete improvement they should expect.

Ans: They should expect better handling of ambiguous words and phrases. Module 9 showed why a fixed Word2Vec lookup cannot change when \

AI Engineering

Q: Why do modern RAG systems overwhelmingly use contextual embedding models rather than static Word2Vec/GloVe-style embeddings, given the added computational cost of contextual models?

Ans: The retrieval quality improvement is substantial and well-justified: this course proved directly, step by step, that static embeddings structurally cannot distinguish different senses of the same word (Module 9), while contextual representations genuinely can (verified in this module).

For a RAG system, this translates to fewer irrelevant retrievals and better handling of naturally ambiguous language — a meaningful accuracy improvement that generally outweighs the added computational cost of contextual embedding computation, especially given that modern hardware and optimized embedding models (covered in the Transformers course) have made this cost increasingly manageable in production.

18. Next Step

Next: Module 14 — Tokenization for Modern NLP — the specific sub-word tokenization schemes (BPE, WordPiece, SentencePiece) that real LLMs use, and the direct bridge to how LLM APIs actually process text.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed