The Context Problem | TechByteByByte

Begin with the central question

If bank has one stored vector, how can it mean money and a river edge?

Essential words

A static embedding always returns the same word vector. Context is surrounding language. A word sense is one particular meaning of an ambiguous word.

What You Will Understand

A direct, numerically verified demonstration of static word embeddings’ fundamental limitation — the one flagged but not yet proven in Module 8: a single word gets exactly one vector, no matter which of its possible meanings is intended. This module proves it concretely, and sets up the shift toward sequence models that can actually use surrounding context.

same word + different sentence -> different intended meaning

Why One Vector per Word Is Not Enough

Module 8 fixed TF-IDF’s total lack of semantic structure — genuinely solving the synonym problem. But it flagged one remaining gap: static embeddings still assign exactly one vector per word. This module exists to make that limitation completely concrete, not just asserted, and to establish the exact principle the rest of this course builds toward: a word’s representation should depend on the words around it.

Meaning Changes with Its Neighbors

a static embedding for “bank” is trained on every sentence that ever used the word — financial sentences and riverbank sentences alike, all mixed together. The resulting vector ends up being a kind of average, or compromise, across every sense it ever saw — good at representing “bank” in general, but not specifically tuned to either specific meaning when it actually matters.

Analogy: The Platypus Word (The Compromise Hybrid) Imagine a biologist who is forced to document all species using a single drawing:

The Problem: The biologist sees a beaver (representing financial banks, vaults, money) and a duck (representing riverbanks, grass, rivers). Because the system forces a single, fixed animal entry per spelling name, the biologist compromises by drawing a Platypus (a static embedding vector).

The Failure: The platypus drawing contains half-beaver, half-duck features.

When a chef requests a duck to roast (a riverbank context), you hand them the platypus. They complain that it has beaver fur.

When a logger requests a beaver to cut trees (a financial context), you hand them the platypus. They complain it has a duck bill.

Because static embeddings retrieve the exact same platypus vector every single time the word “bank” appears, it is a permanent, context-blind compromise. It cannot specialize to its surroundings.

📊 Visual Chart: The Static Embedding Compromise Vector

Here is the geometric layout showing how the single trained static vector sits equidistant between its two pure semantic meanings in vector space:

graph TD
    classDef compromise fill:#f9f,stroke:#333,stroke-width:2px;

PureFinancial["Pure Financial Sense<br>[1.0, 0.0]<br>(Vaults, money, loans)"]
    PureRiverbank["Pure Riverbank Sense<br>[0.0, 1.0]<br>(Water, soil, fishing)"]

Compromise["Static Vector: 'bank'<br>[0.707, 0.707]<br>(Equidistant average)"]:::compromise

Compromise -->|Similarity: 0.707| PureFinancial
    Compromise -->|Similarity: 0.707| PureRiverbank

Context1["Sentence 1:<br>'deposited cash at the bank'"] -->|Retrieves| Compromise
    Context2["Sentence 2:<br>'sat by the muddy bank'"] -->|Retrieves| Compromise

4. Core Concept

"I deposited money in the bank."       <- financial sense

"I sat beside the river bank."          <- riverbank sense

BOTH sentences, using a STATIC embedding, get the EXACT SAME
vector for "bank" -- regardless of which sense is clearly
intended by the surrounding words.

Term	Definition
Static embedding	A word embedding that is fixed once trained — the same vector every time that word appears, regardless of context (Word2Vec, GloVe — Module 8)
Contextual representation	A representation that varies based on the surrounding words — the same word can produce different vectors in different sentences (Module 13)

5. How It Works — Step by Step

1. During Word2Vec/GloVe training (Module 8), "bank" appears in
   BOTH financial contexts ("money", "loan", "account") AND
   riverbank contexts ("water", "fish", "river")
2. The training process has NO way to separate these -- it sees
   ONE word string, "bank", and learns ONE embedding for it
3. The resulting embedding ends up positioned somewhere BETWEEN
   what a "pure financial bank" embedding and a "pure riverbank"
   embedding would look like -- a compromise, not a specialization
4. Every future sentence using "bank" -- regardless of its
   actual intended meaning -- retrieves this SAME compromise
   vector

6. Mathematical Intuition

If a “pure financial bank” embedding and a “pure riverbank” embedding are computed separately (as if each sense were trained in isolation), the actual static embedding (trained on both mixed together) should end up roughly equidistant, in cosine similarity, from both pure senses — demonstrated precisely below.

7. Simple Example

Imagine two hypothetical, “pure” embeddings: one for “bank” if it only ever meant a financial institution, another if it only ever meant a riverbank. The real, static embedding — trained on a corpus containing both uses — should land somewhere between these two pure versions, reasonably similar to each, but a perfect specialization of neither.

8. Build It in Python

What the code will demonstrate

We will construct two clear sense vectors—financial bank and river bank—and then average them to imitate the compromise a static embedding can learn from mixed usage.

The exact coordinates are invented for teaching. The important observation is structural: the same static vector must be returned in both sentences, even when the intended meanings differ.

import numpy as np

financial_context_words = ["money", "loan", "account", "deposit", "interest"]
river_context_words = ["water", "fish", "river", "shore", "sand"]

# Simulated context word embeddings (hand-constructed for illustration)
context_embeddings = {
    "money": np.array([0.9, 0.1]), "loan": np.array([0.85, 0.15]),
    "account": np.array([0.88, 0.05]), "deposit": np.array([0.92, 0.1]),
    "interest": np.array([0.8, 0.2]),
    "water": np.array([0.1, 0.9]), "fish": np.array([0.15, 0.85]),
    "river": np.array([0.05, 0.92]), "shore": np.array([0.1, 0.88]),
    "sand": np.array([0.2, 0.8]),
}

# A STATIC "bank" embedding, trained on BOTH kinds of context mixed together
all_bank_contexts = financial_context_words + river_context_words
bank_static_embedding = np.mean([context_embeddings[w] for w in all_bank_contexts], axis=0)
print("Static 'bank' embedding (averaged across ALL its uses):", np.round(bank_static_embedding, 4))

# What a PURELY financial or PURELY riverbank "bank" would look like
bank_financial_only = np.mean([context_embeddings[w] for w in financial_context_words], axis=0)
bank_river_only = np.mean([context_embeddings[w] for w in river_context_words], axis=0)

print("If 'bank' ONLY meant finance:", np.round(bank_financial_only, 4))
print("If 'bank' ONLY meant river:  ", np.round(bank_river_only, 4))

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

print("\nHow similar is the STATIC (mixed) embedding to each pure sense?")
print("  Similarity to PURE financial sense:", round(cosine_similarity(bank_static_embedding, bank_financial_only), 4))
print("  Similarity to PURE river sense:    ", round(cosine_similarity(bank_static_embedding, bank_river_only), 4))

print("\n'I deposited money at the bank' -- needs the FINANCIAL sense")
print("'We sat on the bank of the river' -- needs the RIVER sense")
print("Both sentences use the EXACT SAME static 'bank' vector:", np.round(bank_static_embedding, 4))

Expected Output:

Static 'bank' embedding (averaged across ALL its uses): [0.495 0.495]
If 'bank' ONLY meant finance: [0.87 0.12]
If 'bank' ONLY meant river:   [0.12 0.87]

How similar is the STATIC (mixed) embedding to each pure sense?
  Similarity to PURE financial sense: 0.7971
  Similarity to PURE river sense:     0.7971

'I deposited money at the bank' -- needs the FINANCIAL sense
'We sat on the bank of the river' -- needs the RIVER sense
Both sentences use the EXACT SAME static 'bank' vector: [0.495 0.495]

9. How It Works

The static embedding lands at [0.495, 0.495] — almost exactly halfway between the “pure financial” ([0.87, 0.12]) and “pure river” ([0.12, 0.87]) embeddings — a genuine, direct numerical compromise.
Its cosine similarity to both pure senses is exactly the same (0.7971) — mathematically equidistant, confirming it’s specialized toward neither sense specifically, exactly the “jack of all senses, master of none” limitation this module set out to prove.
Both example sentences — one clearly financial, one clearly about a river — would retrieve the identical static vector, despite a human reader instantly disambiguating which sense is intended from the surrounding words (“deposited money” vs. “sat…of the river”).

10. Real-World Example

A search or classification system relying on static embeddings for “bank” cannot distinguish a query about “opening a bank account” from one about “walking along the river bank” — both retrieve the same underlying vector representation for the ambiguous word, even though their surrounding context makes the intended meaning completely clear to a human. This is a genuine, practical failure mode in real systems using only static embeddings.

11. The Natural Next Question

“The representation of a word should depend on the surrounding words.”

This is the principle the rest of this course builds toward. A model that could look at “I deposited money at the ___” and produce a different representation for “bank” than it would for “I sat beside the river ___” would directly solve this proven limitation. This is precisely what sequence models (Module 10 — RNNs) were introduced to enable, and what contextual representations (Module 13) and, eventually, attention (Module 12) fully deliver.

12. How Is This Used in Modern AI?

🤖 How Is This Used in Modern AI?

This exact limitation is why modern embedding models used in RAG and semantic search (built on Transformer architectures, covered in the dedicated Transformers course) produce contextual embeddings, not static ones — the same word can and does produce different vectors depending on its sentence, directly solving what this module just proved static embeddings cannot.

Real systems you can recognize

A modern sentence-embedding model processes the complete query or passage before producing its vector, so nearby words can influence the result. OpenAI lists semantic search as an embedding use case, and Hugging Face lists sentence similarity as a dedicated task: OpenAI embeddings and Hugging Face tasks.

GPT and Gemini also compute token representations from surrounding context inside Transformer layers. Their final answer can still misunderstand ambiguity, so context sensitivity must not be confused with guaranteed correctness.

13. How Is This Used in Agentic AI?

Direct relevance to Agentic AI: High, as direct motivation. A RAG system’s retrieval quality depends on correctly distinguishing queries and documents that share ambiguous words but mean different things — exactly the failure mode proven here. This is precisely why production RAG systems use contextual (Transformer-based) embedding models, not static Word2Vec-style embeddings, for retrieval.

14. Common Mistakes / Misunderstandings

⚠️ Mistake: assuming a larger or better-trained static embedding model would fix this. It wouldn’t — the limitation is structural (one vector per word, full stop), not a matter of training data quantity or quality. No amount of additional training data changes the fact that one fixed vector cannot represent multiple, genuinely different meanings simultaneously.

⚠️ Mistake: thinking this is a rare, edge-case problem. Polysemy (Module 1) is pervasive in ordinary language — this isn’t a corner case, it’s a systematic gap affecting a meaningful fraction of common words.

⚠️ Mistake: assuming the fix is “train separate embeddings for each word sense.” This isn’t practical or how the field actually solved it — the real fix (Modules 10-13) is architectural: models that incorporate surrounding context dynamically, producing a representation on the fly rather than looking up a fixed one.

15. Important Distinctions

Static Embedding (Module 8)	Contextual Representation (Module 13)
ONE fixed vector per word	A DIFFERENT vector per word, depending on context
Trained once, looked up afterward	Computed dynamically, incorporating surrounding words
Proven directly: cannot distinguish “bank” senses	Designed specifically to distinguish them

16. When to Use

Static embeddings remain a reasonable, lightweight choice when word-sense ambiguity isn’t a major concern for the task, or when compute/latency constraints favor a simpler, faster representation over a full contextual model.

17. When Not to Use

Don’t rely on static embeddings for tasks where word-sense disambiguation is important — search, question-answering, or anything requiring precise semantic understanding of ambiguous terms will suffer from exactly the limitation proven directly in this module.

18. Production Considerations

Modern production embedding models are contextual (Transformer- based), specifically because of this proven limitation — a genuinely important reason to prefer them over legacy static embeddings for any serious semantic search or RAG application.
This limitation is invisible until tested specifically — a system using static embeddings might appear to work well on many queries, while silently failing on the specific subset involving ambiguous words, making this a good candidate for targeted evaluation test cases.

19. Interview Questions

Beginner

Q: What is the core limitation of static word embeddings?

Ans: A static embedding assigns exactly one fixed vector to a word, regardless of the context it appears in — so a word like “bank,” which can mean either a financial institution or a riverbank, gets only one representation that has to serve both meanings, rather than a representation specific to whichever sense is actually intended in a given sentence.

Intermediate

Q: Why does a static embedding for an ambiguous word like “bank” end up resembling neither of its individual senses particularly well?

Ans: Because the embedding is trained on ALL the word’s uses mixed together — both financial and riverbank contexts contribute to shaping the same single vector. The result is a kind of average or compromise position, demonstrated directly in this module: the static embedding was mathematically equidistant (identical cosine similarity) from what a “pure financial” and “pure riverbank” embedding would each look like — reasonably related to both, but a precise match for neither.

Advanced

Q: Why can’t simply training a static embedding model on more data fix the word-sense ambiguity problem?

Ans: The limitation isn’t about insufficient training data — it’s structural. A static embedding model, by definition, produces exactly one vector per unique word string, regardless of how much data it’s trained on. More training data would refine that one vector’s position based on the AGGREGATE of all the word’s uses, but it fundamentally cannot produce multiple different vectors for the same word string based on context, since the architecture has no mechanism for context-dependent representation at inference time.

Fixing this requires a different kind of model architecture entirely — one that computes a word’s representation dynamically, incorporating its actual surrounding context each time (sequence models, Module 10, and ultimately attention, Module 12).

Scenario

Q: A team’s search system uses static embeddings and performs well overall, but a targeted evaluation reveals it consistently confuses queries about “bank accounts” with results about “riverbanks.” What’s the underlying cause, and what’s the fix?

Ans: This is a direct, real-world instance of the limitation proven in this module — the static embedding for “bank” represents a compromise between its financial and riverbank senses, so queries and documents using either sense end up positioned similarly in the embedding space, regardless of which sense is actually intended. The genuine fix isn’t more training data or tuning — it requires switching to contextual embeddings (Module 13), which produce a different representation for “bank” depending on its surrounding words, correctly distinguishing “bank account” from “riverbank” based on context.

AI Engineering

Q: Why do virtually all modern production embedding models (used for RAG, semantic search) use contextual, Transformer-based architectures rather than static Word2Vec/GloVe-style embeddings?

Ans: Static embeddings have a proven, structural limitation: they assign exactly one vector per word, unable to distinguish different senses of ambiguous words based on context — demonstrated directly in this module, where the same “bank” vector had to serve both financial and riverbank meanings.

Contextual, Transformer-based embedding models solve this by computing a word’s (or, more precisely, a token’s or sentence’s) representation dynamically, incorporating the actual surrounding context — this directly improves retrieval quality for exactly the kind of ambiguous queries this module demonstrated static embeddings mishandling, which is why they’re the standard choice for serious production semantic search and RAG systems today.

20. Next Step

Next: Module 10 — NLP with RNNs — the first architectural answer to “a word’s representation should depend on its surrounding words”: sequential processing with a hidden state carrying context forward.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026