TechByteByByte

Bag of Words

Build the Bag of Words representation by hand and confirm it with scikit-learn — vocabulary, document-term matrix, binary vs. frequency counts — and a direct proof of its core limitation: it knows words exist, but not their order or meaning.

#NLP#AI#Bag of Words#Document-Term Matrix

Begin with the central question

How can different-length documents become equal-sized rows of numbers?

Essential words

A vocabulary lists words being counted. A document-term matrix has one row per document and one column per vocabulary word. A feature is one numerical model input.

What You Will Understand

The first genuinely useful numerical text representation: Bag of Words. You’ll build a document-term matrix entirely by hand, confirm it matches scikit-learn’s implementation exactly, and directly prove its defining limitation — it has no sense of word order at all.

Document -> count each vocabulary word -> fixed-length vector

Why Documents Need Fixed-Size Features

Module 2 got you to token IDs — a sequence of numbers per document, but sequences of different documents have different lengths, which most classical ML models (your ML course) can’t directly handle as input (they generally expect fixed-size feature vectors). Bag of Words exists to solve this: convert every document into a fixed-length vector, regardless of the document’s length, by counting word occurrences.

A Document Becomes Word Counts

imagine dumping all the words from a document into a bag, losing all sense of their original order — then just counting how many of each word type ended up in the bag. Two documents with the same words, in different orders, would produce identical bags — which is exactly the trade-off “Bag of Words” makes: simplicity and fixed-size vectors, at the direct cost of ignoring order entirely.

Analogy: The Scrambled Ingredient Soup Imagine you are trying to guess what dish is being prepared in a kitchen by looking only at a bag containing all the used wrappers and scrap ingredients:

  • The Representation: You open the bag and count: “flour: 2 bags, sugar: 1 cup, egg: 3 shells”.
  • The Limitation: You have absolutely no idea what order these ingredients were mixed in, what temperature they were baked at, or what the final structure looks like. You just have the grocery counts.
  • For simple categorization (e.g. distinguishing a dessert from a salad), this ingredient count alone is often more than enough — a dessert has high sugar/flour counts, while a salad has high lettuce/tomato counts.
  • But for fine-grained understanding (e.g., distinguishing a cake recipe from a cookie recipe using the exact same ingredients), the lack of order instructions (syntax) makes it impossible to solve.

📊 Visual Chart: Document-Term Matrix Alignment

Here is how documents of varying lengths are compiled into a unified matrix of a fixed, vocabulary-sized width:

graph TD
    Doc1["Doc 1:<br>'cat eats fish'"] --> Vector1["Vector 1:<br>[ cat: 1, dog: 0, eats: 1, fish: 1, sleeps: 0 ]"]
    Doc2["Doc 2:<br>'dog eats fish'"] --> Vector2["Vector 2:<br>[ cat: 0, dog: 1, eats: 1, fish: 1, sleeps: 0 ]"]
    Doc3["Doc 3:<br>'cat sleeps'"] --> Vector3["Vector 3:<br>[ cat: 1, dog: 0, eats: 0, fish: 0, sleeps: 1 ]"]

subgraph DTM ["Unified Document-Term Matrix (DTM)"]
        Row1["Row 1: [ 1, 0, 1, 1, 0 ]"]
        Row2["Row 2: [ 0, 1, 1, 1, 0 ]"]
        Row3["Row 3: [ 1, 0, 0, 0, 1 ]"]
    end

Vector1 --> Row1
    Vector2 --> Row2
    Vector3 --> Row3

4. Core Concept

TermDefinition
VocabularyEvery unique word across all documents (Module 2)
Document-term matrixA table where each row is a document, each column is a vocabulary word, and each cell is that word’s count in that document
Frequency representationCell values are word counts
Binary representationCell values are just 0/1 — “does this word appear at all?”
Document 1: "cat eats fish"
Document 2: "dog eats fish"

Vocabulary: [cat, dog, eats, fish]

Document-term matrix:
              cat  dog  eats  fish
Document 1:    1    0    1     1
Document 2:    0    1    1     1

5. How It Works — Step by Step

1. Build the VOCABULARY across all documents (Module 2)
2. For each document, create a vector the SAME LENGTH as the
   vocabulary
3. For each vocabulary word, count how many times it appears in
   THIS document -- that count becomes this vector's value at
   that word's position
4. Stack all documents' vectors into a DOCUMENT-TERM MATRIX

6. Mathematical Intuition

No formula beyond simple counting — every vector’s length equals the vocabulary size, and entry i is count(vocabulary[i] in this document). The one thing worth being precise about: every document’s vector has the exact same length, regardless of how long the original document was — this fixed-size property is precisely why Bag of Words vectors work as input to classical ML models (Module 6).

7. Simple Example

For vocabulary [cat, dog, eats, fish], the document “cat cat eats fish fish fish” produces the frequency vector [2, 0, 1, 3] — “cat” appears twice, “dog” zero times, “eats” once, “fish” three times. The binary version of the same document would instead be [1, 0, 1, 1] — just “does this word appear at all,” discarding the actual counts too.

8. Build It in Python

What the code will demonstrate

We will create the vocabulary first and then count how often each vocabulary word appears in each document. Every vector position has a fixed meaning because it corresponds to one vocabulary entry.

Pay special attention to two outputs: the vector length stays fixed for every document, while changing word order without changing counts leaves the vector unchanged.

Before you run it

This example uses scikit-learn. Install it once in the same Python environment with pip install scikit-learn. If ModuleNotFoundError: No module named 'sklearn' appears, the package is missing; the NLP logic has not run yet.

# --- Bag of Words, built entirely by hand ---
doc1 = "cat eats fish"
doc2 = "dog eats fish"
documents = [doc1, doc2]

tokenized = [doc.split() for doc in documents]
vocabulary = sorted(set(word for doc in tokenized for word in doc))
print("Vocabulary:", vocabulary)

def bow_vector(tokens, vocab):
    return [tokens.count(word) for word in vocab]

vectors = [bow_vector(doc, vocabulary) for doc in tokenized]
print("\nDocument-term matrix (rows=documents, columns=vocabulary):")
for doc, vec in zip(documents, vectors):
    print(f"  '{doc}': {vec}")

# --- Confirm with scikit-learn ---
from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer()
sklearn_matrix = vectorizer.fit_transform(documents)
print("\nscikit-learn vocabulary:", sorted(vectorizer.vocabulary_.keys()))
print("scikit-learn document-term matrix:\n", sklearn_matrix.toarray())

# --- Binary vs frequency representation ---
doc3 = "cat cat eats fish fish fish"
tokens3 = doc3.split()
freq_vec = bow_vector(tokens3, vocabulary)
binary_vec = [1 if count > 0 else 0 for count in freq_vec]
print(f"\nDocument: '{doc3}'")
print("Frequency vector:", freq_vec)
print("Binary vector:   ", binary_vec)

# --- The core limitation: word order is completely ignored ---
doc_a = "cat eats fish"
doc_b = "fish eats cat"   # opposite meaning, SAME bag of words
vec_a = bow_vector(doc_a.split(), vocabulary)
vec_b = bow_vector(doc_b.split(), vocabulary)
print(f"\n'{doc_a}' -> {vec_a}")
print(f"'{doc_b}' -> {vec_b}")
print("Identical vectors despite opposite meaning?", vec_a == vec_b)

Expected Output:

Vocabulary: ['cat', 'dog', 'eats', 'fish']

Document-term matrix (rows=documents, columns=vocabulary):
  'cat eats fish': [1, 0, 1, 1]
  'dog eats fish': [0, 1, 1, 1]

scikit-learn vocabulary: ['cat', 'dog', 'eats', 'fish']
scikit-learn document-term matrix:
 [[1 0 1 1]
 [0 1 1 1]]

Document: 'cat cat eats fish fish fish'
Frequency vector: [2, 0, 1, 3]
Binary vector:    [1, 0, 1, 1]

'cat eats fish' -> [1, 0, 1, 1]
'fish eats cat' -> [1, 0, 1, 1]
Identical vectors despite opposite meaning? True

9. How It Works

  • The hand-built vectors exactly match scikit-learn’s CountVectorizer output — [[1,0,1,1],[0,1,1,1]] both times — confirming Bag of Words is genuinely this simple a counting operation, not hidden complexity.
  • The core limitation is proven directly, not just asserted: “cat eats fish” and “fish eats cat” — grammatically and semantically very different sentences (one implies the cat is the eater; the other implies the fish is) — produce exactly identical Bag of Words vectors (True). This is the concrete, numeric version of “Bag of Words knows that words exist, but not much about meaning or order.”

10. Strengths and Weaknesses

Strengths

  • Extremely simple to compute and understand.
  • Produces fixed-size vectors, directly usable by classical ML models.
  • Works reasonably well for tasks where word presence matters more than order (e.g., basic topic classification).

Weaknesses

  • Word order is completely discarded — proven directly above.
  • Every word is treated as equally important — common words like “the” get the same weighting mechanism as rare, informative words (Module 5’s TF-IDF directly addresses this).
  • Vectors are large and sparse — most documents use only a tiny fraction of the full vocabulary, so most vector entries are zero.
  • No sense of semantic similarity — “cat” and “dog” are just as numerically unrelated as “cat” and “spaceship” in this representation (Module 8’s embeddings directly address this).

11. How Is This Used in Modern AI?

🤖 How Is This Used in Modern AI?

Bag of Words is rarely the final representation in modern AI systems, but its core idea — representing text as a fixed-size numerical vector — is the conceptual ancestor of every text representation this course covers, up to and including embeddings (Module 8).

Use caseBag of Words’ role today
Quick baselinesA fast, simple baseline to compare more sophisticated models against
Simple keyword-based filtersStill genuinely useful for basic presence/absence-of-terms checks
Educational/interpretabilityDirectly interpretable — you can see exactly which words drove a prediction

Real systems you can recognize

Scikit-learn still provides CountVectorizer, which converts documents into sparse token-count matrices. It remains a practical baseline for spam, routing, and topic classifiers where speed and interpretability matter. See scikit-learn text feature extraction.

GPT and Gemini do not use Bag of Words internally: their token order is essential. A production application may nevertheless place a cheap count-based filter before an LLM to handle a narrow, high-volume rule or classification task.

12. How Is This Used in Agentic AI?

Direct relevance to Agentic AI: Low, directly. Modern agent systems use embeddings (Module 8) and Transformer-based models, not raw Bag of Words, for semantic understanding. Its value here is entirely foundational — understanding this simplest possible representation, and precisely why it falls short (word order, semantic similarity), is what makes every subsequent representation in this course (TF-IDF, embeddings, contextual representations) feel like a well-motivated improvement rather than an arbitrary alternative.

13. Common Mistakes / Misunderstandings

⚠️ Mistake: assuming Bag of Words captures any word order information. As proven directly, it provably does not — two sentences with opposite meanings can produce identical vectors.

⚠️ Mistake: assuming all words are treated with equal genuine importance for a task. They ARE treated equally in raw Bag of Words — this is itself a limitation Module 5’s TF-IDF exists to address.

⚠️ Mistake: assuming Bag of Words vectors capture any notion of word meaning. “Cat” and “dog” are just as numerically distant as “cat” and any other unrelated word in this representation — no semantic relationship is encoded at all.

14. Important Distinctions

Frequency RepresentationBinary Representation
Cell value = word countCell value = 0 or 1 (present or not)
Captures how OFTEN a word appearsOnly captures WHETHER a word appears
Bag of WordsTF-IDF (Module 5)
Every word weighted equallyRare, informative words weighted more heavily
Simple countingCounting adjusted by how common a word is across documents

15. When to Use

Use Bag of Words for quick baselines, simple keyword-presence checks, or genuinely small-scale, interpretable classical NLP tasks where word order and semantic similarity aren’t essential to the task.

16. When Not to Use

Don’t use raw Bag of Words for tasks where word order or meaning genuinely matters (which is most real language understanding tasks) — its provable blindness to order and lack of semantic understanding make it a poor fit whenever these matter, motivating every subsequent module in this course.

17. Production Considerations

  • Vector sparsity — for a vocabulary of tens of thousands of words, most document vectors will be almost entirely zeros; sparse matrix representations (like scikit-learn’s default output format) are essential for memory efficiency at scale.
  • Vocabulary size growth — as more documents are processed, vocabulary (and therefore vector length) can grow very large; some systems cap vocabulary size to the most frequent N words as a practical constraint.

18. Interview Questions

Beginner

Q: What is Bag of Words?

Ans: A text representation technique that converts a document into a fixed-length numerical vector by counting how many times each vocabulary word appears in it — the document’s original word order is discarded entirely.

Intermediate

Q: What’s the difference between a frequency-based and a binary Bag of Words representation?

Ans: A frequency representation records how many times each vocabulary word appears in a document (its actual count). A binary representation only records whether each word appears at all (1) or not (0), discarding the actual count information — useful when presence/absence matters more than exact frequency for a given task.

Advanced

Q: Prove, conceptually, why Bag of Words cannot distinguish between sentences with the same words but different, even opposite, meanings.

Ans: Bag of Words constructs a vector purely by counting word occurrences against a fixed vocabulary — the position of each count in the vector corresponds to a vocabulary word, not to any position within the original sentence. Since “cat eats fish” and “fish eats cat” contain exactly the same set of words with exactly the same counts, their resulting vectors are mathematically identical, regardless of the completely different (in this case, arguably opposite) meaning implied by their different word orders — demonstrated directly, where both sentences produced the exact same vector.

Scenario

Q: A team builds a Bag of Words-based classifier to detect customer complaints, and it performs poorly on sentences like “I would NOT recommend this, it’s terrible” versus “I would recommend this, it’s not terrible” — treating them very similarly. What’s happening?

Ans: Both sentences share almost the exact same set of words (“recommend,” “terrible,” “not,” etc.) — Bag of Words has no way to capture that the placement of “not” completely reverses the sentiment in each case, since it only counts word occurrences without regard to order or which words “not” is modifying. This is a direct, practical instance of Bag of Words’ proven blindness to word order — a task like this, where negation placement genuinely changes meaning, needs a representation that captures more than just word presence, such as embeddings or a sequence-aware model (Modules 8-13).

AI Engineering

Q: Why would Bag of Words be a reasonable first baseline when building a new text classification system, even knowing its limitations?

Ans: It’s extremely fast and cheap to compute, fully interpretable (you can directly see which words correlate with which predictions), and often achieves surprisingly reasonable performance on tasks where word presence alone carries substantial signal (like basic topic classification).

Establishing this simple baseline first gives you a concrete performance number to compare more sophisticated approaches (TF-IDF, embeddings, Transformer-based models) against — if a much more complex approach doesn’t meaningfully beat the Bag of Words baseline, that’s valuable information about whether the added complexity is actually justified for your specific task.

19. Next Step

Next: Module 5 — TF-IDF — directly addressing Bag of Words’ “every word weighted equally” limitation, with a full worked formula.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed