Begin with the problem
Before an AI system can search a document, it must collect it, identify it, and track where it came from. Ingestion creates that trustworthy starting record.
source → parse → chunk → attach metadata → indexWhat you will learn
- Explain Documents & Ingestion in simple language before using its technical details.
- Follow the mechanism step by step through a small RAG example.
- Connect this topic to the modules before and after it.
- Decide when to use it, when not to use it, and what to measure in production.
Current real-system grounding: Google’s File Search guide shows how a current managed system imports files, creates chunks and embeddings, stores them, and carries retrieval metadata.
The product example proves that the pattern is used in a real system. It does not mean every provider uses the same hidden algorithm, defaults, limits, or pricing.
1. The problem this module solves
Module 4 named “Document Loading” as the first offline pipeline stage without unpacking it. Before discussing embeddings or chunking, it’s worth asking a really foundational question: what exactly are we retrieving, and what does “loading” a document actually involve? Getting this stage right shapes everything downstream.
2. The Problem — Real Knowledge Lives in Many Different Forms
A real company’s knowledge isn’t one uniform format. It’s really scattered across:
- PDFs (policy documents, contracts, reports)
- Word documents
- Markdown files (engineering docs, READMEs)
- HTML (internal wikis, web pages)
- Databases (structured records)
- Emails
- JSON / CSV (exported data)
- APIs (live data from other systems)
- Knowledge bases (Confluence, Notion)
- Code repositoriesA RAG system needs to really handle this diversity — a pipeline
that only handles clean .txt files is useless for the messy reality
of most organizations’ actual documentation.
3. Ingestion — The First Real Step
Document (in its ORIGINAL format, wherever it lives)
↓
LOADER (format-specific logic for pulling out raw content)
↓
Raw Content (extracted, but not yet cleaned or structured)A loader’s job is narrow and specific: get the content OUT of its original container, reliably, along with enough information about WHERE it came from to be useful later.
Notice this is deliberately separate from parsing (Module 6) — a loader’s job is to retrieve the raw content; a parser’s job is to make sense of that content’s structure. Keeping these concerns distinct makes the pipeline easier to reason about and debug.
4. Why Metadata Really Matters — Captured at Ingestion Time
This is the part of ingestion that’s easy to underestimate. Every document should carry metadata alongside its content:
document_id
source (which system/file this came from)
page (for multi-page documents)
section (which part of the document)
author
created_date
last_modified_date
department
permissions (who is allowed to see this)Why capture this NOW, at ingestion, rather than later? Because this information often only exists at the SOURCE — the file system knows the last-modified date; the wiki knows which team owns the page; the HR system knows which document is HR-only. Once content is extracted and mixed into a general text pipeline, this contextual information is really difficult or impossible to recover later. Capture it now, or lose it.
5. What Metadata Enables Later
FILTERING (Module 15): "only search HR documents" -- a
really faster, more precise search
than searching everything
CITATIONS (Module 23): "this answer came from Section 4.2
of the 2025 Travel Policy, page 3" --
really impossible without metadata
captured here
ACCESS CONTROL (Module 27): "don't show this employee HR-only
documents" -- a real security
requirement that depends on
permission metadata existing
RANKING: "prefer more RECENT
documents" -- needs a
created_date to work
DEBUGGING: "why did the system
retrieve THIS chunk?" --
source metadata makes
this traceableEvery one of these later capabilities traces directly back to decisions made at this very first ingestion stage.
6. A Real Developer Example
TechCorp ingests documents from THREE different sources:
1. A PDF policy document from the HR file share
2. A Confluence wiki page about engineering on-call rotations
3. A CSV export of the current employee directory
Even though these are WILDLY different formats, EACH loader's job is
the SAME: extract raw content + capture source metadata.
PDF loader captures: document_id, page numbers, PDF filename
Wiki loader captures: document_id, page URL, last-edited
timestamp, owning team
CSV loader captures: document_id (per row, perhaps),
source file, column headers
Later, when an employee asks "who's on call this week?", the system
can FILTER to only engineering-sourced documents, and correctly CITE
the Confluence page as the source -- because that metadata was
captured HERE, at ingestion, not invented later.7. A Simple Agentic AI Connection
An agent with a “fetch document” tool relies on this exact ingestion concept — when it pulls a document into its context, it should really capture (or be given) the same kind of source metadata, so that if the agent later cites information in its response, that citation traces back to a real, identifiable source rather than being an unattributed claim.
8. How Is This Used in AI?
🤖 How Is This Used in AI?
Every production RAG system’s ingestion layer is built around format-specific loaders (a PDF loader, an HTML loader, a database connector) that each solve the narrow problem of extracting content and metadata from one specific source type, feeding into a unified downstream pipeline (parsing, chunking, embedding) that doesn’t need to know or care which original format the content came from.
9. Real-World Applications
- Enterprise knowledge bases spanning wikis, file shares, and databases
- Customer support systems ingesting help articles, past ticket resolutions, and product documentation together
- Legal and compliance systems where document provenance (source, author, date) is a real, non-negotiable requirement
10. Common Mistakes
Incorrect idea: Extracting only text and discarding metadata at ingestion time.
Why it is incorrect: As shown directly in Section 4, this information is often unrecoverable later — capture it now.
Incorrect idea: Building one monolithic loader that tries to handle every format.
Why it is incorrect: As shown directly in Section 6, format-specific loaders that share a common output structure are really more maintainable.
Incorrect idea: Treating ingestion and parsing as the same step.
Why it is incorrect: As emphasized directly in Section 3, keeping “extract raw content” separate from “make sense of structure” (Module 6) is a really useful architectural boundary.
11. Limitations
- Ingestion can only capture metadata that really exists at the source — if a source system doesn’t track authorship or permissions, that information simply isn’t available to capture
- Different source systems have wildly different metadata richness — some really require additional manual tagging to achieve the metadata quality Section 5’s capabilities depend on
12. Quick Reference — The Whole Idea in One Diagram
Document (original format, original location)
↓
Format-specific LOADER
↓
Raw Content + METADATA (document_id, source, page, author, date,
permissions...)
↓
Feeds into: Parsing (Module 6) -> Chunking (Module 7-8) -> Embedding
(Module 10)
Metadata captured HERE enables: filtering, citations, access control,
ranking, debugging -- LATER13. Code — Building Format-Aware Loaders With Metadata
🎯 Target of this example: implement Section 6’s real developer example directly — really different loader functions for different source types, each producing a UNIFORM output structure carrying both content and metadata, demonstrating Section 5’s downstream benefits.
Example 1 — Simple
from dataclasses import dataclass
from datetime import datetime
@dataclass
class LoadedDocument:
"""A UNIFORM structure every loader produces, regardless of the
original source format -- exactly Section 8's 'downstream doesn't
need to know the original format' principle."""
document_id: str
content: str
source: str
metadata: dict
def load_text_document(document_id: str, raw_text: str, source_path: str) -> LoadedDocument:
"""The simplest possible loader -- plain text, minimal metadata."""
return LoadedDocument(
document_id=document_id, content=raw_text, source=source_path,
metadata={"format": "text", "loaded_at": datetime.now().isoformat()}
)
doc = load_text_document(
"travel_policy_v3", "International hotel reimbursement is limited to $200/night.",
"/hr_share/travel_policy.txt"
)
print(f"Document ID: {doc.document_id}")
print(f"Content: {doc.content}")
print(f"Metadata: {doc.metadata}")Expected Output:
Document ID: travel_policy_v3
Content: International hotel reimbursement is limited to $200/night.
Metadata: {'format': 'text', 'loaded_at': '2026-08-19T14:22:10.512384'}What we conclude from this example: even this minimal loader
already separates content from metadata — exactly Section 3’s
principle that a loader’s job is to extract content AND capture
provenance information alongside it, not just the text alone.
Example 2 — Intermediate
from dataclasses import dataclass
from datetime import datetime
@dataclass
class LoadedDocument:
document_id: str
content: str
source: str
metadata: dict
def load_pdf_document(document_id: str, raw_text: str, filename: str, page_count: int) -> LoadedDocument:
"""A PDF-specific loader -- captures PDF-specific metadata
(page_count) that a plain text loader wouldn't have."""
return LoadedDocument(
document_id=document_id, content=raw_text, source=filename,
metadata={"format": "pdf", "page_count": page_count,
"loaded_at": datetime.now().isoformat()}
)
def load_wiki_page(document_id: str, raw_text: str, url: str, owning_team: str, last_edited: str) -> LoadedDocument:
"""A wiki-specific loader -- captures WIKI-specific metadata
(owning_team, last_edited) that a PDF loader wouldn't have."""
return LoadedDocument(
document_id=document_id, content=raw_text, source=url,
metadata={"format": "wiki", "owning_team": owning_team,
"last_edited": last_edited, "loaded_at": datetime.now().isoformat()}
)
pdf_doc = load_pdf_document(
"hr_travel_policy", "International hotel reimbursement is limited to $200/night.",
"travel_policy_2026.pdf", page_count=12
)
wiki_doc = load_wiki_page(
"eng_oncall_rotation", "The on-call rotation follows a weekly schedule starting Monday.",
"https://wiki.techcorp.com/oncall", owning_team="Platform Engineering", last_edited="2026-08-15"
)
for label, doc in [("PDF document", pdf_doc), ("Wiki document", wiki_doc)]:
print(f"{label}: source={doc.source}")
print(f" Metadata: {doc.metadata}\n")Expected Output:
PDF document: source=travel_policy_2026.pdf
Metadata: {'format': 'pdf', 'page_count': 12, 'loaded_at':
'2026-08-19T14:22:11.203847'}
Wiki document: source=https://wiki.techcorp.com/oncall
Metadata: {'format': 'wiki', 'owning_team': 'Platform Engineering',
'last_edited': '2026-08-15', 'loaded_at':
'2026-08-19T14:22:11.203912'}What we conclude from this example: each loader captures metadata
really SPECIFIC to its source type (page_count for PDFs,
owning_team for wikis) — exactly Section 6’s real developer example,
demonstrating that format-specific loaders can enrich metadata based
on what’s actually available at each source, while still producing the
same uniform LoadedDocument structure downstream stages can rely on.
Example 3 — Production Grade
from dataclasses import dataclass, field
from datetime import datetime
from enum import Enum
class SourceFormat(Enum):
TEXT = "text"
PDF = "pdf"
WIKI = "wiki"
CSV = "csv"
@dataclass
class LoadedDocument:
document_id: str
content: str
source: str
format: SourceFormat
permissions: list = field(default_factory=lambda: ["all_employees"])
extra_metadata: dict = field(default_factory=dict)
loaded_at: str = field(default_factory=lambda: datetime.now().isoformat())
class DocumentIngestionPipeline:
"""A production-style ingestion pipeline ROUTING to the correct
loader based on format, and ENFORCING that permissions are always
captured -- directly implementing Section 5's access-control
downstream benefit, made structural rather than optional."""
def load(self, document_id: str, raw_content: str, source: str,
format: SourceFormat, permissions: list = None, **extra) -> LoadedDocument:
return LoadedDocument(
document_id=document_id, content=raw_content, source=source, format=format,
permissions=permissions or ["all_employees"], extra_metadata=extra,
)
def load_batch(self, raw_documents: list) -> list:
loaded = []
for doc in raw_documents:
loaded.append(self.load(**doc))
return loaded
pipeline = DocumentIngestionPipeline()
raw_documents = [
{"document_id": "hr_travel_policy", "raw_content": "Hotel reimbursement limited to $200/night.",
"source": "travel_policy_2026.pdf", "format": SourceFormat.PDF,
"permissions": ["all_employees"], "page_count": 12},
{"document_id": "salary_bands_2026", "raw_content": "Engineering salary bands: L3 $95k-$130k.",
"source": "salary_bands.xlsx", "format": SourceFormat.CSV,
"permissions": ["hr_only", "management"], "row_count": 45},
]
loaded_docs = pipeline.load_batch(raw_documents)
for doc in loaded_docs:
print(f"[{doc.document_id}] format={doc.format.value}, permissions={doc.permissions}")
print(f" Extra metadata: {doc.extra_metadata}\n")Expected Output:
[hr_travel_policy] format=pdf, permissions=['all_employees']
Extra metadata: {'page_count': 12}
[salary_bands_2026] format=csv, permissions=['hr_only',
'management']
Extra metadata: {'row_count': 45}What we conclude from this example: permissions is enforced as a
required, always-present field (defaulting to all_employees rather
than being silently omitted) — this is exactly the kind of structural
guarantee Section 5’s access-control benefit needs: permission
metadata that MUST be captured at ingestion time, not left as an
afterthought that could be forgotten for a really sensitive
document like salary bands.
14. Interview Questions
Q: What is the specific job of a document loader in a RAG pipeline, and how does that differ from parsing?
Ans: A loader’s job is to extract raw content from a document’s original format and location, and capture metadata about its source — where it came from, when it was created, who owns it, what permissions apply. Parsing, by contrast, is about making sense of that raw content’s internal structure — headers, tables, sections. Keeping these as separate concerns means a loader’s job stays simple and format-specific (a PDF loader vs. a wiki loader), while parsing logic can be shared or specialized independently.
Q: Why is it important to capture metadata like permissions and source information at ingestion time, rather than adding it later in the pipeline?
Ans: This information often only exists at the original source system — the file system knows who owns a file, the HR system knows which documents are HR-only. Once content has been extracted and mixed into a general text-processing pipeline, this contextual information can be really difficult or impossible to reconstruct. Capturing it at ingestion, while the connection to the original source is still direct, ensures it’s available for everything downstream that depends on it — filtering, citations, and access control.
Q: Give three concrete downstream capabilities that depend directly on metadata captured during ingestion.
Ans: Metadata filtering (searching only within a specific department’s documents, which requires department metadata), citations (telling a user exactly which document and section an answer came from, which requires source and page metadata), and access control (preventing an employee from seeing HR-only content, which requires permission metadata) — all three are really impossible to implement reliably if the relevant metadata wasn’t captured when the document was first ingested.
Q: Why might a real system use different loader functions for PDFs, wiki pages, and CSV files, rather than one universal loader?
Ans: Different source formats have really different available metadata and extraction challenges — a PDF loader can capture page counts and needs to handle PDF-specific parsing quirks, while a wiki loader can capture the owning team and last-edited timestamp that a PDF simply doesn’t have. Format-specific loaders let each one capture the richest metadata actually available for its source type, while still producing a uniform output structure that the rest of the pipeline (chunking, embedding) can process without needing to know which original format the content came from.
15. What You Should Remember
- Ingestion extracts raw content AND captures source metadata — really separate from parsing, which makes sense of internal structure.
- Metadata captured now — source, permissions, dates, ownership — enables filtering, citations, and access control later, and is often unrecoverable if not captured at this stage.
- Format-specific loaders producing a uniform output structure is the standard, practical pattern — verified directly through a production pipeline that structurally enforces permission metadata capture for every document.
16. Quick Practice
Design the metadata fields you’d want to capture when ingesting a customer support ticket resolution into a RAG knowledge base (distinct from a policy PDF) — what’s really specific to this source type that a generic loader might miss?
17. Next Step
Next: Module 6 — Document Parsing — why extracting usable text from real documents (especially PDFs with tables, columns, and images) is really harder than it looks, and why poor parsing can silently doom an otherwise well-designed RAG pipeline.
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed