Chunking

The RAG article referenced “retrieved chunks” without explaining how documents actually get broken into pieces in the first place. This article covers exactly that: chunking.

The simple definition

Chunking is the process of breaking a large document into smaller, more manageable pieces before storing and indexing it for retrieval. Recall from the Vector Database article’s discussion of storing documents alongside their embeddings — a chunk is the actual unit that gets converted into one embedding and stored, rather than embedding an entire, lengthy document as one single, undifferentiated block.

Why documents can’t just be embedded whole

Recall from the Embedding article that an embedding captures the overall meaning of whatever text it’s applied to.

A single embedding for an entire 50-page document would have to compress everything in that document — dozens of distinct topics and facts — into one vector, badly blurring together information that a specific query might only need a small piece of.

Chunking solves this by breaking documents into smaller, more topically focused pieces, each with its own embedding, letting retrieval find precisely the specific paragraph relevant to a question rather than an entire, overly broad document.

flowchart LR
    A[Full 50-page document] --> B[Chunking: split into smaller pieces]
    B --> C[Chunk 1: Embedding 1]
    B --> D[Chunk 2: Embedding 2]
    B --> E["...many more chunks"]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of indexing a large reference book not by creating one single card for the entire book, but by creating a separate index card for each individual section or paragraph — letting someone searching the index find the exact relevant page rather than being pointed to the whole book and having to search through it themselves.

Where this breaks down: A librarian creating index cards applies genuine judgment about natural topic boundaries. Chunking, in most real systems, is a largely mechanical process — splitting text based on rules like character count or paragraph breaks, covered fully in the Chunk Size article next, rather than a human manually deciding where each natural topical boundary genuinely falls.

Why chunking strategy is a genuinely consequential engineering decision

This is worth grounding in real, published findings, not treating chunking as a trivial implementation detail.

Published research on RAG systems has specifically identified chunking as a critical, and historically underexplored, design decision — poorly formed chunks, cutting off in the middle of a relevant idea or mixing unrelated content together, force the generator to work with noisy, incomplete context, a documented contributor to the hallucination and omission problems covered throughout this phase.

Common chunking approaches

Real chunking strategies range from simple to sophisticated. Fixed-size chunking splits text into pieces of a set character or token count, simple and fast but sometimes cutting sentences or ideas awkwardly in half. Paragraph or section-based chunking splits along a document’s natural structural boundaries, better preserving coherent ideas but producing chunks of inconsistent size.

Semantic chunking, a more advanced, actively researched approach, uses the meaning of the text itself to decide where natural topic boundaries fall, aiming to keep genuinely related content together even when it doesn’t align with simple paragraph breaks.

A concrete example, layered

For a simple beginner example: chunking a company’s 20-page HR policy document into roughly paragraph-sized pieces means a question about vacation days retrieves just the vacation policy paragraph, not the entire 20-page document.

For a production example: Amazon Bedrock Knowledge Bases, referenced in the Knowledge Base article, offers configurable chunking strategies as a core part of its document ingestion pipeline, reflecting how central this specific decision is treated in real, deployed enterprise RAG products.

Watch one document become chunks

Document
├── Heading: Returns
│   ├── General return window
│   └── Damaged products
├── Heading: Non-returnable products
│   ├── Opened hygiene products
│   └── Personalized products
└── Heading: Refund timing

A structure-aware chunker can preserve these sections instead of cutting after an arbitrary character count. If “Opened hygiene products cannot be returned” is separated from the Non-returnable products heading, the chunk loses useful meaning.

Common chunking strategies

Strategy	Advantage	Limitation
Fixed token or character count	Simple and predictable	Can cut through sentences or sections.
Paragraph-based	Preserves prose boundaries	Paragraphs can vary greatly in size.
Structure-aware	Uses headings, lists, tables, or code units	Requires reliable document parsing.
Semantic	Splits when topic meaning changes	More computation and tuning are required.

Tables, source code, scanned PDFs, and conversations often need specialized chunkers. A single chunking rule rarely performs best for every content type.

What Google actually does during ingestion

When a document enters Vertex AI RAG Engine, Google runs transformations that prepare it for indexing, including splitting it into chunks. A real application might ingest a 60-page handbook once, create hundreds of searchable chunks, and reuse those chunks for thousands of questions.

Google also supports a Document AI Layout Parser, which can preserve useful document structure instead of treating every PDF as one unbroken string. This matters for headings and tables whose meaning can be damaged by a careless cut. See Google’s RAG transformation documentation.

Common misconception

A frequent beginner assumption: that chunking is a minor, largely automatic preprocessing step that doesn’t meaningfully affect final answer quality. As the published research cited above demonstrates, this significantly understates its importance — chunking strategy is a genuine, high-leverage design decision that directly shapes what retrieval can find and, by extension, what the model can accurately answer.

Where this fits in what comes next

You now understand why and how documents get broken into pieces before retrieval. The next article, Chunk Size, covers the specific, practical question of exactly how large each of those pieces should be — a genuine trade-off with real consequences for retrieval accuracy.

In one sentence

Chunking breaks documents into smaller, retrievable pieces before indexing, and published research has established it as a genuinely consequential design decision — poorly formed chunks directly degrade retrieval accuracy and contribute to hallucination, making this deceptively simple step far more important than it first appears.

Related Terms

• Retrieval-Augmented Generation (RAG)
• Embedding
• Chunk Size
• Chunk Overlap
• Retrieval