Chunk Size

The Chunking article covered breaking documents into pieces without specifying exactly how big those pieces should be. This article covers precisely that decision: chunk size.

The simple definition

Chunk size is the target length of each individual chunk, typically measured in tokens or characters, that a document gets split into during chunking. A chunk size of 500 tokens means each piece of a chunked document is roughly that length — recall from the Token article that this is the same unit used throughout language model pricing and context limits.

Why this specific number is a genuine trade-off, not an arbitrary setting

Recall from the Embedding article that a vector’s meaning comes from the overall content it represents.

A small chunk size produces highly focused, precise chunks — a retrieval match is very likely to be genuinely relevant, since there’s little room for unrelated content to dilute it — but small chunks can lack enough surrounding context for a model to fully understand what they mean in isolation, and a single idea might get awkwardly split across multiple separate chunks.

A large chunk size preserves more context and coherence within a single chunk, but risks diluting the embedding’s focus — mixing several distinct topics into one vector, exactly the “blurred meaning” problem the Chunking article described whole-document embedding causing, just at a smaller scale.

flowchart LR
    A[Small chunk size] --> B[Precise matches, but less context, ideas may split awkwardly]
    C[Large chunk size] --> D[More context per chunk, but less precise, diluted matches]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of photographing a large mural. Zooming in tightly on one small section gives you sharp, precise detail of that one specific part, but you lose the surrounding context needed to understand what that section is actually depicting as part of the whole. Zooming out to capture the entire mural in one shot preserves the full context, but any single small detail becomes harder to make out clearly.

Where this breaks down: A photograph’s zoom level is a simple, continuous physical adjustment.

Chunk size interacts with the specific mechanics of the embedding model doing the encoding, as covered in the Embedding Model article — different embedding models have their own practical limits and behave differently at different input lengths, meaning the “right” chunk size genuinely depends on which specific embedding model a system is actually using, not just a universal preference for more or less context.

How this choice actually gets made in practice

This connects directly to real, practical decisions engineers make when building a RAG system, as covered in the RAG article. A knowledge base of short, distinct FAQ entries might use a small chunk size, since each entry is already naturally focused.

A knowledge base of long, narrative legal contracts might use a larger chunk size, since legal clauses often depend heavily on surrounding context to be correctly interpreted.

There’s no single universally correct number — 500 tokens is a commonly cited starting point in practice, but real systems are tuned experimentally, using the retrieval evaluation techniques covered later in this phase.

A concrete example, layered

For a simple beginner example: chunking a cookbook with a chunk size of 100 tokens might split a single recipe’s ingredient list from its instructions, hurting retrieval when someone searches for a dish by its cooking method described only in the instructions section.

For a production example: Amazon Bedrock Knowledge Bases, referenced throughout this phase, exposes chunk size as a directly configurable parameter for developers building a knowledge base, letting a company tune this setting specifically for their own document types rather than being locked into one fixed default.

Why getting this wrong has real, measurable consequences

It’s worth connecting this directly back to the Chunking article’s published research finding: poorly formed chunks degrade generation quality and contribute to hallucination.

Chunk size is one of the most direct levers controlling exactly how “well-formed” a chunk actually is — too small, and a chunk may be retrieved without enough context for the model to use it correctly; too large, and the chunk’s embedding may be too diluted for retrieval to find it as a relevant match in the first place.

Compare three chunk sizes

Suppose a 1,200-token policy contains six short sections:

Chunk size 100  → about 12 chunks; precise but context may fragment
Chunk size 300  → about 4 chunks; useful middle ground for this example
Chunk size 1,200→ 1 chunk; complete document but much irrelevant material

A small chunk can retrieve the exact sentence about refund timing, while a larger chunk may preserve the exceptions written in a neighboring paragraph. Neither size is universally correct.

Choose candidate sizes, run them on representative documents, and evaluate whether relevant evidence appears in the top results. Also measure final-answer quality, latency, token usage, and duplication rather than optimizing chunk size by intuition alone.

Verified production numbers from Google

Vertex AI RAG Engine documents a default chunk_size of 1,024 tokens. That is a real service default, not a universal recommendation. Google explains that smaller chunks can produce more precise embeddings, while larger chunks preserve more surrounding information but can blur details. Source: Google’s RAG transformations guide.

Common misconception

A frequent beginner assumption: that there’s one universally correct chunk size that works well for every knowledge base and every task. As this article has shown, the right chunk size depends genuinely on the specific content type, the embedding model in use, and the kinds of questions the system needs to answer — a setting that real teams tune experimentally for their specific use case, not a fixed, one-size-fits-all default.

Where this fits in what comes next

You now understand the size trade-off chunking involves. The next article, Chunk Overlap, covers a closely related technique that helps mitigate chunk size’s biggest weakness — the risk of an important idea getting awkwardly split right at a chunk boundary.

In one sentence

Chunk size is the target length of each chunk, and choosing it well means balancing precision (smaller chunks, more focused matches) against context (larger chunks, more coherent meaning) — a real, experimentally-tuned decision with direct, measurable consequences for retrieval accuracy.

Related Terms

• Chunking
• Token
• Embedding Model
• Chunk Overlap
• Retrieval Evaluation