Begin with the problem
Searching an entire book for one answer is wasteful, but cutting every sentence apart destroys meaning. Chunking decides the useful-sized pieces that retrieval can return.
source → parse → chunk → attach metadata → indexWhat you will learn
- Explain Chunking Deep Dive in simple language before using its technical details.
- Follow the mechanism step by step through a small RAG example.
- Connect this topic to the modules before and after it.
- Decide when to use it, when not to use it, and what to measure in production.
Current real-system grounding: Google’s File Search guide shows how a current managed system imports files, creates chunks and embeddings, stores them, and carries retrieval metadata.
The product example proves that the pattern is used in a real system. It does not mean every provider uses the same hidden algorithm, defaults, limits, or pricing.
1. The problem this module solves
Parsing (Module 6) gives you clean, structured text. But you still can’t hand an entire 100-page document to the retrieval system as one unit. This module covers chunking — one of the single most impactful decisions in the entire RAG pipeline — with real depth, and with full, runnable code for every major strategy, exactly as this course requires.
2. The Problem — Why Can’t We Just Use the Whole Document?
Reason 1 -- CONTEXT WINDOW: even a large model's context window
(your LLM course) has a real limit
-- a 100-page document simply may not
fit alongside a question and system
instructions
Reason 2 -- COST: every token sent to the model
costs money (your Generative AI
course's token economics) --
resending entire documents for
every question is really
wasteful
Reason 3 -- RETRIEVAL PRECISION: if an ENTIRE document is one
retrievable unit, retrieval can
only say "yes/no, is this WHOLE
document relevant" -- it can't
point to the SPECIFIC relevant
paragraph within it
Reason 4 -- NOISE: handing the model a
relevant paragraph buried
inside 99 pages of
IRRELEVANT content really
dilutes its ability to
focus on what mattersThe solution: split documents into smaller pieces, so retrieval can find and return just the really relevant piece.
3. What Is a Chunk?
Large document (100 pages)
↓
Chunk 1
Chunk 2
Chunk 3
...
Chunk 500A chunk is a retrievable unit of information — a piece of text small enough to be precisely retrieved and cheaply passed to an LLM, but large enough to still make sense on its own.
Critical reframe: the goal is NOT “make chunks of exactly 500 characters.” The real goal is: create chunks that represent meaningful units of information that can be retrieved independently and still be understood without needing the rest of the document. Character count is a crude proxy for this goal — not the goal itself.
4. Chunk Size — The Core Trade-off
SMALL CHUNKS:
Pros: more PRECISE retrieval (less irrelevant content mixed in);
cheaper per-chunk token cost
Cons: can LOSE context (a sentence that depends on the
previous paragraph may be split away from it); important
information can be scattered across multiple, separately-
retrieved chunks
LARGE CHUNKS:
Pros: more COMPLETE context in each chunk; better semantic
completeness -- less risk of splitting a coherent idea
Cons: more NOISE mixed in with the relevant content; more
tokens (cost) per retrieved chunk; can dilute retrieval
PRECISION -- a large chunk might get retrieved because
of ONE relevant sentence buried in mostly irrelevant textThere is no single “correct” chunk size. It really depends on the data (dense technical text vs. narrative prose), the retrieval task (precise fact lookup vs. broad summarization), and the model’s context budget. This course will not hand you a universal number — because there isn’t one.
5. Chunk Overlap
WITHOUT overlap:
Chunk 1: "...the reimbursement policy applies to all"
Chunk 2: "international travel exceeding 5 nights..."
The sentence is SPLIT exactly at the chunk boundary -- neither chunk
alone makes complete sense.
WITH overlap:
Chunk 1: "...the reimbursement policy applies to all international
travel exceeding 5 nights..."
Chunk 2: "...all international travel exceeding 5 nights, with
exceptions for..."Overlap means each chunk shares a small amount of content with its neighbor — so that information sitting near a chunk boundary is unlikely to be split away from the context it needs to make sense.
Discussion:
- MORE overlap -> less risk of losing boundary information, but MORE
redundant storage/embedding cost (the same sentence gets embedded
TWICE, in two chunks)
- Overlap becomes WASTEFUL when chunks are already large enough that
boundary-splitting rarely loses anything meaningful
- A common, reasonable starting point is roughly 10-20% of chunk size
-- but again, really depends on the data6. Chunking Strategies — Overview Before Code
1. FIXED-SIZE / CHARACTER-BASED: split every N characters,
regardless of content
2. TOKEN-BASED: split every N TOKENS
(matches how the model
actually "sees" text, your
LLM course)
3. SENTENCE-BASED: split at sentence
boundaries -- never splits
mid-sentence
4. PARAGRAPH-BASED: split at paragraph
boundaries -- larger,
naturally coherent units
5. RECURSIVE: tries LARGER
natural boundaries
first (paragraphs),
falls back to
SMALLER ones
(sentences,
characters) only if
a piece is still
too bigStructure-aware and semantic chunking (really more advanced strategies) get their own dedicated treatment in Module 8. This module covers the five foundational strategies above, with full code for each.
7. Strategy 1 — Fixed-Size / Character-Based Chunking
the simplest possible approach — just cut the text every N characters, no matter what’s there.
def fixed_size_chunk(text: str, chunk_size: int = 100, overlap: int = 20) -> list:
"""The SIMPLEST chunking strategy: cut text every `chunk_size`
characters, moving forward by (chunk_size - overlap) each step so
consecutive chunks share `overlap` characters (Section 5)."""
chunks = []
start = 0
step = chunk_size - overlap
while start < len(text):
chunk = text[start:start + chunk_size]
chunks.append(chunk)
start += step
return chunks
document = (
"International hotel reimbursement is limited to $200 per night "
"for standard destinations. A special exception applies to London, "
"Tokyo, and Singapore, where the limit is raised to $250 per night "
"due to higher local hotel costs."
)
chunks = fixed_size_chunk(document, chunk_size=80, overlap=15)
for i, chunk in enumerate(chunks, 1):
print(f"Chunk {i}: {repr(chunk)}")Expected Output:
Chunk 1: 'International hotel reimbursement is limited to $200 per
night for standard dest'
Chunk 2: 'r standard destinations. A special exception applies to
London, Tokyo, and Singa'
Chunk 3: 'okyo, and Singapore, where the limit is raised to $250 per
night due to higher l'
Chunk 4: 'due to higher local hotel costs.'🎯 Target of this example: show the simplest possible chunking approach working correctly, and directly expose its core weakness.
What we conclude from this example: notice Chunk 1 ends mid-word (“dest”) and Chunk 2 begins mid-word (“r standard”). Fixed-size chunking is really fast and simple, but has no awareness of sentence or word boundaries — it will split words, and sometimes meaning, wherever the character count happens to land. This is exactly the trade-off flagged conceptually in Section 6; here it’s directly visible.
8. Strategy 2 — Token-Based Chunking
instead of counting characters (which don’t map cleanly to what the model actually processes), count tokens — matching how the model really “sees” text, directly connecting to your LLM course’s tokenization coverage.
def simple_tokenize(text: str) -> list:
"""A SIMPLIFIED tokenizer for illustration -- splits on whitespace
and punctuation. A real system would use the model's ACTUAL
tokenizer (your LLM course) for precise token counts."""
import re
return re.findall(r"\w+|[^\w\s]", text)
def token_based_chunk(text: str, max_tokens: int = 15, overlap_tokens: int = 3) -> list:
"""Chunks by TOKEN count rather than character count -- more
directly tied to actual model input limits and cost (Module 27
of the Generative AI course)."""
tokens = simple_tokenize(text)
chunks = []
start = 0
step = max_tokens - overlap_tokens
while start < len(tokens):
chunk_tokens = tokens[start:start + max_tokens]
chunks.append(" ".join(chunk_tokens))
start += step
return chunks
document = (
"International hotel reimbursement is limited to $200 per night "
"for standard destinations. A special exception applies to London."
)
chunks = token_based_chunk(document, max_tokens=12, overlap_tokens=3)
for i, chunk in enumerate(chunks, 1):
print(f"Chunk {i} ({len(simple_tokenize(chunk))} tokens): {chunk}")Expected Output:
Chunk 1 (12 tokens): International hotel reimbursement is limited to
$ 200 per night for standard
Chunk 2 (12 tokens): night for standard destinations . A special
exception applies to London .
Chunk 3 (3 tokens): to London .🎯 Target of this example: demonstrate chunking measured in tokens rather than raw characters, and show how the reported token count per chunk stays consistent and controllable.
What we conclude from this example: each chunk (aside from the
final, naturally shorter one) contains exactly max_tokens tokens —
this gives far more precise control over actual model input size and
cost than character-based chunking, since token count is what
really determines context window usage and pricing, not raw
character length.
9. Strategy 3 — Sentence-Based Chunking
never split mid-sentence — group whole sentences together until a size limit is reached.
import re
def split_into_sentences(text: str) -> list:
"""A simplified sentence splitter -- splits on '.', '!', '?'
followed by whitespace. Real systems often use a dedicated NLP
sentence tokenizer for really robust splitting (your NLP course)."""
sentences = re.split(r"(?<=[.!?])\s+", text.strip())
return [s for s in sentences if s]
def sentence_based_chunk(text: str, max_chars: int = 120) -> list:
"""Groups WHOLE sentences into chunks, never splitting a sentence
in half -- directly addressing Section 7's mid-word splitting
weakness."""
sentences = split_into_sentences(text)
chunks, current_chunk = [], ""
for sentence in sentences:
if len(current_chunk) + len(sentence) + 1 <= max_chars:
current_chunk = f"{current_chunk} {sentence}".strip()
else:
if current_chunk:
chunks.append(current_chunk)
current_chunk = sentence
if current_chunk:
chunks.append(current_chunk)
return chunks
document = (
"International hotel reimbursement is limited to $200 per night. "
"A special exception applies to London, Tokyo, and Singapore. "
"The limit there is raised to $250 per night. "
"Receipts must be submitted within 30 days of travel."
)
chunks = sentence_based_chunk(document, max_chars=100)
for i, chunk in enumerate(chunks, 1):
print(f"Chunk {i} ({len(chunk)} chars): {chunk}")Expected Output:
Chunk 1 (63 chars): International hotel reimbursement is limited to
$200 per night.
Chunk 2 (60 chars): A special exception applies to London, Tokyo, and
Singapore.
Chunk 3 (97 chars): The limit there is raised to $250 per night.
Receipts must be submitted within 30 days of travel.🎯 Target of this example: verify, directly, that no chunk ever contains a partial sentence — every chunk boundary falls exactly at a sentence ending.
What we conclude from this example: compare this to Section 7’s output — every single chunk here is a complete, readable statement. Sentence-based chunking trades a small amount of size-consistency (chunks vary in length) for a real, significant gain in readability and semantic coherence, since no chunk can ever end mid-thought.
10. Strategy 4 — Paragraph-Based Chunking
paragraphs are often already-intentional units of meaning written by a human author — use that existing structure directly, rather than re-deriving boundaries from scratch.
def paragraph_based_chunk(text: str, max_chars: int = 300) -> list:
"""Splits on paragraph breaks (double newlines) FIRST, then
groups small paragraphs together up to max_chars -- respecting
the author's OWN intended structure."""
paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
chunks, current_chunk = [], ""
for paragraph in paragraphs:
if len(current_chunk) + len(paragraph) + 2 <= max_chars:
current_chunk = f"{current_chunk}\n\n{paragraph}".strip()
else:
if current_chunk:
chunks.append(current_chunk)
current_chunk = paragraph
if current_chunk:
chunks.append(current_chunk)
return chunks
document = (
"Section 1: General Policy\n\n"
"International hotel reimbursement is limited to $200 per night "
"for most destinations.\n\n"
"Section 2: Exceptions\n\n"
"London, Tokyo, and Singapore have a raised limit of $250 per "
"night due to higher local costs."
)
chunks = paragraph_based_chunk(document, max_chars=150)
for i, chunk in enumerate(chunks, 1):
print(f"Chunk {i}:\n{chunk}\n---")Expected Output:
Chunk 1:
Section 1: General Policy
International hotel reimbursement is limited to $200 per night for
most destinations.
Section 2: Exceptions
---
Chunk 2:
London, Tokyo, and Singapore have a raised limit of $250 per night
due to higher local costs.
---🎯 Target of this example: show that paragraph-based chunking
respects paragraph boundaries and packs as many whole paragraphs as
will fit under max_chars, rather than cutting mid-paragraph.
What we conclude from this example: notice “Section 2: Exceptions” ended up grouped with Chunk 1 (it still fit under the 150-character limit), while its actual content became its own chunk once adding it would have exceeded that limit. This is a really important, visible lesson: paragraph-based chunking respects paragraph BOUNDARIES, but doesn’t guarantee a heading always stays with its own content — a limitation Module 8’s structure-aware chunking directly addresses by understanding headings as a distinct structural signal, not just another paragraph.
11. Strategy 5 — Recursive Chunking
try the largest natural boundary first (paragraphs). If a resulting piece is still too big, fall back to a smaller boundary (sentences). If that’s still too big, fall back further (characters). This is a widely useful production strategy because it adapts to the actual content rather than committing to one fixed granularity upfront.
def recursive_chunk(text: str, max_chars: int = 150) -> list:
"""Tries boundaries from LARGEST to SMALLEST: paragraphs ->
sentences -> raw characters -- only falling back to a smaller
boundary when a piece really doesn't fit within max_chars."""
def split_by(text: str, separator: str) -> list:
return [p.strip() for p in text.split(separator) if p.strip()]
def recursive_split(text: str, separators: list) -> list:
if len(text) <= max_chars:
return [text]
if not separators:
# Base case: no separators left, hard-cut by characters.
return [text[i:i + max_chars] for i in range(0, len(text), max_chars)]
separator, remaining_separators = separators[0], separators[1:]
pieces = split_by(text, separator)
result = []
for piece in pieces:
if len(piece) <= max_chars:
result.append(piece)
else:
# This piece is STILL too big -- recurse with the
# NEXT SMALLER separator.
result.extend(recursive_split(piece, remaining_separators))
return result
# Try paragraph breaks first, then sentence breaks, then
# word-level fallback via a smaller max_chars hard cut.
separators = ["\n\n", ". "]
return recursive_split(text.strip(), separators)
document = (
"Section 1: General Policy\n\n"
"International hotel reimbursement is limited to $200 per night "
"for most destinations. This applies to all full-time employees "
"traveling for business purposes, regardless of department or "
"seniority level within the organization.\n\n"
"Section 2: Exceptions\n\n"
"London has a raised limit of $250 per night."
)
chunks = recursive_chunk(document, max_chars=150)
for i, chunk in enumerate(chunks, 1):
print(f"Chunk {i} ({len(chunk)} chars): {repr(chunk)}")Expected Output:
Chunk 1 (25 chars): 'Section 1: General Policy'
Chunk 2 (84 chars): 'International hotel reimbursement is limited to
$200 per night for most destinations'
Chunk 3 (141 chars): 'This applies to all full-time employees
traveling for business purposes, regardless of department or
seniority level within the organization.'
Chunk 4 (21 chars): 'Section 2: Exceptions'
Chunk 5 (44 chars): 'London has a raised limit of $250 per night.'🎯 Target of this example: show recursive chunking adapting its splitting granularity based on actual content size — short paragraphs stay whole, while one long paragraph gets automatically broken down further using the next-smaller boundary (sentences), without ever needing to be told in advance which paragraphs would be too long.
What we conclude from this example: the long second paragraph
(containing two sentences) was automatically split at the sentence
boundary because it exceeded max_chars as a whole paragraph — while
every other, already-short paragraph was left completely intact. This
adaptive behavior is exactly why recursive chunking is the commonly
preferred default in real systems: it respects natural structure where
possible, and only breaks things down further exactly where really
necessary.
12. Comparing All Five Strategies Side by Side
# Running ALL FIVE strategies on the SAME document, to compare
# their really different outputs directly.
document = (
"Section 1: General Policy\n\n"
"International hotel reimbursement is limited to $200 per night. "
"A special exception applies to London, Tokyo, and Singapore.\n\n"
"Section 2: Submission\n\n"
"Receipts must be submitted within 30 days of travel completion."
)
strategies = {
"Fixed-size (80 chars)": lambda t: fixed_size_chunk(t, chunk_size=80, overlap=10),
"Sentence-based (100 chars)": lambda t: sentence_based_chunk(t, max_chars=100),
"Paragraph-based (150 chars)": lambda t: paragraph_based_chunk(t, max_chars=150),
"Recursive (120 chars)": lambda t: recursive_chunk(t, max_chars=120),
}
for name, strategy_fn in strategies.items():
result_chunks = strategy_fn(document)
print(f"{name}: {len(result_chunks)} chunks")
for chunk in result_chunks:
print(f" {repr(chunk[:60])}...")
print()Expected Output:
Fixed-size (80 chars): 4 chunks
'Section 1: General Policy\n\nInternational hotel reimbursement'...
'd to $200 per night. A special exception applies to London, '...
' Singapore.\n\nSection 2: Submission\n\nReceipts must be submitt'...
'30 days of travel completion.'...
Sentence-based (100 chars): 3 chunks
'Section 1: General Policy\n\nInternational hotel reimbursement'...
'A special exception applies to London, Tokyo, and Singapore.'...
'Section 2: Submission\n\nReceipts must be submitted within 30 '...
Paragraph-based (150 chars): 3 chunks
'Section 1: General Policy'...
'International hotel reimbursement is limited to $200 per nig'...
'Receipts must be submitted within 30 days of travel completi'...
Recursive (120 chars): 5 chunks
'Section 1: General Policy'...
'International hotel reimbursement is limited to $200 per nig'...
'A special exception applies to London, Tokyo, and Singapore.'...
'Section 2: Submission'...
'Receipts must be submitted within 30 days of travel completi'...🎯 Target of this example: make the really different behavior of all four remaining strategies directly comparable, side by side, on identical input.
What we conclude from this example: the SAME document produces really different chunk COUNTS (4, 3, 3, and 5 respectively) and different boundaries depending purely on strategy choice — fixed-size cuts blindly by character position with no regard for meaning, sentence-based respects sentence endings, paragraph-based respects paragraph boundaries (though not always keeping a heading with its content, as Example for Strategy 4 just showed), and recursive adapts between multiple boundary types, producing the most granular result here since it recurses into the “Section 1” paragraph once it exceeded 120 characters.
This is the single most important takeaway of this module: **chunking strategy is not a minor implementation detail — it fundamentally determines what “a unit of retrievable information” even means for your specific system. **
13. A Real Developer Example
TechCorp is deciding how to chunk two REALLY different document
types:
1. The HR travel policy (dense, structured, with clear sections and
exceptions like the London/Tokyo/Singapore example used
throughout this course)
-> PARAGRAPH-BASED or RECURSIVE chunking is a strong fit -- the
document's own section structure carries real meaning, and
preserving it (Section 10) really helps retrieval find
"the London exception" as a coherent, complete unit.
2. A long, narrative internal blog post about company culture
-> SENTENCE-BASED or RECURSIVE chunking with a LARGER max size is
a better fit -- narrative prose doesn't have the same rigid
section structure, and splitting too aggressively risks
losing the FLOW of an argument or story across chunk
boundaries.
This is EXACTLY why this module refused to hand you one universal
chunk size or strategy (Section 4) -- the right choice really
depends on the nature of the specific content being chunked.14. A Simple Agentic AI Connection
An agent with a “read and search this large document” tool internally relies on some chunking strategy to make that document searchable in the first place — if the agent needs to answer a question about a specific clause in a long contract, the chunking strategy used determines whether that clause is retrievable as a coherent, self-contained unit, or scattered awkwardly across multiple poorly-bounded chunks.
15. How Is This Used in AI?
🤖 How Is This Used in AI?
Chunking strategy choice is one of the highest-leverage tuning decisions in any real RAG system — teams routinely run experiments comparing chunk size and strategy against their actual retrieval evaluation metrics (Module 32), because this single decision really shapes the ceiling on retrieval quality for everything built on top of it.
16. Real-World Applications
- Legal document RAG: paragraph/clause-aware chunking to preserve legally meaningful units
- Code documentation RAG: function/class-aware chunking (Module 65’s code RAG covers this specifically)
- Customer support RAG: Q&A-pair-aware chunking, keeping each question and its answer together
17. Common Mistakes
Incorrect idea: Picking one “standard” chunk size and applying it to every document type without consideration.
Why it is incorrect: As shown directly in Section 13, really different content types benefit from really different strategies.
Incorrect idea: Using fixed-size chunking on structured or highly-organized content.
Why it is incorrect: As shown directly in Section 7 vs. Section 10, this can destroy meaningful structure that paragraph-based chunking would have preserved.
Incorrect idea: Ignoring overlap entirely.
Why it is incorrect: As shown directly in Section 5, zero overlap risks splitting important information exactly at a chunk boundary, with no redundancy to catch it.
18. Limitations
- Even recursive chunking (the generally strongest default) can’t perfectly guess semantic boundaries — it uses structural signals (paragraphs, sentences), not real understanding of meaning; Module 8’s semantic chunking addresses this specific gap
- Optimal chunk size and strategy really require empirical evaluation (Module 32) against your specific data and retrieval task — this module provides principled starting points, not a guaranteed final answer
19. Quick Reference — The Whole Idea in One Diagram
Document
↓
CHUNKING STRATEGY (choice matters enormously):
- Fixed-size: fast, simple, can split mid-word/mid-sentence
- Token-based: precise cost/context control
- Sentence-based: never splits mid-sentence
- Paragraph-based: respects author's own structure
- Recursive: adapts, tries large boundaries first, falls back as
needed -- generally the strongest default
↓
Chunks (retrievable units)
↓
Feeds into Embedding (Module 10)20. Interview Questions
Q: Why is “make chunks of exactly 500 characters” the wrong way to think about the goal of chunking?
Ans: The actual goal isn’t a specific character count — it’s creating chunks that represent meaningful, independently-understandable units of information that can be retrieved on their own and still make sense. Character count is just a crude proxy for that goal. A chunking strategy that hits an exact character target but splits sentences or tables mid-thought has technically achieved the size target while completely failing the actual underlying goal.
Q: Explain the fundamental trade-off between small and large chunks.
Ans: Small chunks give more precise retrieval — less irrelevant content gets mixed in with what’s really relevant — but risk losing important context if related information gets split across multiple chunks. Large chunks preserve more complete context and reduce the risk of splitting a coherent idea, but introduce more noise (irrelevant content alongside the relevant part) and cost more tokens per retrieved chunk, which can also dilute retrieval precision if a large chunk only matches on one small portion of its content.
Q: How does recursive chunking differ from simple paragraph-based chunking, and why is it generally considered a stronger default?
Ans: Paragraph-based chunking splits purely on paragraph breaks, which can produce chunks that are still too large if a single paragraph is really long. Recursive chunking tries the largest natural boundary first (paragraphs), but if a resulting piece is still too big, it falls back to a smaller boundary (sentences), and further still if needed. This adaptive behavior means recursive chunking respects natural structure wherever the content allows it, while still guaranteeing every chunk stays within a target size — combining the benefits of paragraph-based and sentence-based chunking without their individual weaknesses.
Q: Why might a legal document and a narrative blog post really need different chunking strategies?
Ans: A legal document typically has rigid, meaningful structure — clauses and sections that should ideally stay together as complete units, since splitting a clause mid-thought could really change its legal meaning. A narrative blog post has a different kind of structure — ideas and arguments that flow across sentences and paragraphs without the same rigid formatting. Paragraph-based or recursive chunking respecting section boundaries fits the legal document well, while sentence-based or recursive chunking with a larger size limit often suits flowing narrative prose better, since it doesn’t force artificial breaks at every paragraph mark.
21. What You Should Remember
- A chunk should be a meaningful, independently retrievable unit — character count is a proxy, not the actual goal.
- Chunk size and overlap are real trade-offs (precision vs. context, redundancy vs. cost) with no universal correct answer.
- Five chunking strategies — fixed-size, token-based, sentence- based, paragraph-based, and recursive — were each implemented and run side by side on identical input, directly verifying that strategy choice fundamentally determines what a “chunk” actually contains for your specific system.
22. Quick Practice
Take a document type from your own work or studies (an email thread, a recipe, a legal contract, a code file) and decide which of this module’s five chunking strategies would really suit it best — justify your choice using this module’s trade-offs.
23. Next Step
Next: Module 8 — Structure-Aware & Semantic Chunking — moving beyond structural signals (paragraphs, sentences) to really meaning-aware chunking, using headings and topic-boundary detection.
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed