Retrieval-Augmented Generation (RAG)

The Retrieval article covered searching a knowledge base. This article covers the complete system this entire phase has been building toward — combining that search with a language model’s generation: RAG.

The simple definition

RAG is a technique where a language model’s response is grounded in relevant information retrieved from an external knowledge base, rather than relying purely on what the model memorized during training. Recall from the Vector Database article’s earliest mention of RAG, back in the Data Representation phase — this article gives that idea its full, dedicated treatment.

A user’s question gets used to retrieve relevant content, and that content gets fed into the model alongside the original question, letting the model generate an answer grounded in real, specific, retrieved information.

Why this technique was genuinely necessary

Recall from the Pretraining article’s real limitation: a model’s knowledge is frozen at whatever point its training data was collected, and it has no access to private, company-specific information it never saw during training at all.

Recall also from the Next-Token Prediction article’s discussion of hallucination — when a model lacks real information, it can still produce fluent, confident-sounding, but incorrect text. RAG directly addresses both problems by giving the model actual, current, specific source material to draw from at the moment of answering, rather than relying entirely on frozen, general training knowledge.

flowchart LR
    A[User's question] --> B[Retrieval: search Knowledge Base]
    B --> C[Relevant content found]
    A --> D[Combine question + retrieved content]
    C --> D
    D --> E[Language Model generates grounded answer]

The real, published origin of this technique

This deserves to be grounded precisely, since RAG has a specific, well-documented research origin. The technique was formalized in a 2020 paper by Lewis and colleagues, combining a dense passage retriever with a sequence-to-sequence language model — the original academic foundation for the entire pattern covered throughout this phase.

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of an open-book exam versus a closed-book one. A closed-book exam relies entirely on what a student memorized in advance — the equivalent of a raw pretrained model relying only on training data. An open-book exam lets the student consult specific reference material while answering — the equivalent of RAG, where the model consults retrieved, real source documents while generating its response, rather than relying purely on memory.

Where this breaks down: A student consults an open book with genuine reading comprehension and judgment about what’s relevant.

A RAG system’s “consultation” is the mechanical retrieval process covered in the previous article, and the model’s use of that retrieved content is still the same statistical, next-token-prediction process covered throughout the Language Models phase — no genuine comprehension of the source material, just a much better-informed starting point for its predictions.

A concrete example, layered

For a simple beginner example: asked “What’s our company’s remote work policy,” a RAG-powered internal chatbot retrieves the specific relevant paragraph from the company handbook and generates an answer grounded in that actual, current text, rather than guessing based on general knowledge of what companies typically do.

For a production example: this exact pattern — retrieve relevant chunks, feed them to the model, generate a grounded answer — underlies real, deployed products across customer support, legal research, and internal enterprise search, with frameworks like LangChain and LlamaIndex, and managed services like Amazon Bedrock Knowledge Bases (mentioned in the Knowledge Base article), all providing standardized tooling specifically for building this pipeline.

Why RAG genuinely helps, and what it doesn’t fully solve

It’s worth being honest about a real, documented limitation, not presenting RAG as a complete fix for hallucination.

RAG substantially reduces hallucination by giving the model real source material to ground its answer in, but it doesn’t eliminate the problem entirely — if retrieval returns irrelevant or incomplete content, as covered in the Retrieval article’s discussion of published research, the model can still generate an inaccurate answer, or even fail to properly use the good context it was given.

RAG shifts and reduces the risk; it doesn’t remove it, a distinction covered fully in the Grounding and Hallucination articles closing out this phase.

A real story: what happens when a company skips this

This is worth telling in full, because it’s a genuine, landmark legal case, not a hypothetical warning. In February 2024, a Canadian tribunal ruled against Air Canada after its website chatbot told a grieving customer, Jake Moffatt, that he could book a flight immediately and claim a bereavement discount within 90 days after travel.

That was wrong — Air Canada’s actual policy required the discount to be requested before booking.

When Moffatt tried to claim his refund, the airline refused, then argued in court that it couldn’t be held responsible for its own chatbot’s words, calling it a “separate legal entity responsible for its own actions.” The tribunal called that argument “remarkable” and ordered Air Canada to pay the refund anyway, noting the airline “did not take reasonable care to ensure its chatbot was accurate.” This is exactly the failure mode RAG, done well, is built to prevent: the chatbot’s answer wasn’t grounded in Air Canada’s actual, current bereavement policy page — it was generated confidently, without being properly tied back to the real source of truth sitting right there on the airline’s own website.

The complete RAG request

flowchart TD
    A[User question] --> B[Rewrite or expand query if needed]
    B --> C[Retrieve candidate chunks]
    C --> D[Apply permissions and metadata filters]
    D --> E[Rerank the best evidence]
    E --> F[Build prompt with question and sources]
    F --> G[LLM generates grounded answer]
    G --> H[Return answer with citations]

Suppose an employee asks, “How many parental-leave weeks do I receive in India?” The system retrieves the current India policy, excludes policies for other countries, and gives the relevant passage to GPT, Gemini, or another model. The model then writes a clear answer based on that passage and cites the source.

RAG does not train the model

The retrieved text enters the model’s current context. It does not update model weights during a normal request, which means a corrected policy can be indexed and used immediately without fine-tuning.

OpenAI’s Responses API can use file search, and its vector-store search API returns relevant chunks with attributes and scores. Gemini supports grounding and retrieval patterns through its APIs and Google Cloud services. These managed features implement parts of RAG, while applications still remain responsible for source quality, authorization, evaluation, and answer behavior.

When RAG is the wrong tool

Do not add RAG when the model already performs the stable task reliably and no external knowledge is needed. Do not use it as a substitute for a transactional database query when an exact account balance or inventory count must be obtained through a structured tool.

A realistic Gemini application flow

Suppose a retailer builds a Gemini assistant for its current return policy. A customer asks, “Can I return opened headphones after 20 days?” The application retrieves the latest policy passages, places them beside the question, and asks Gemini to answer only from that evidence.

Customer question
      ↓
Vertex AI Search or RAG Engine retrieves policy passages
      ↓
Question + passages are sent to Gemini
      ↓
Gemini writes an answer and the application displays its sources

The policy is supplied at request time; it was not permanently learned by Gemini. Google documents this pattern in its APIs for search and RAG experiences.

Common misconception

A frequent beginner assumption: that RAG involves retraining or fine-tuning the language model on the knowledge base’s content.

As this article has explained, this isn’t accurate — RAG works entirely at inference time, as covered in the Inference article, simply adding retrieved text to the model’s input sequence for that specific request; the model’s own weights never change, and the knowledge base can be updated instantly without any retraining at all, a major practical advantage over fine-tuning a model on new information directly.

Where this fits in what comes next

You now understand the complete RAG pattern this phase is built around. The next several articles zoom into the specific engineering decisions that make a real RAG system actually work well — starting with Chunking, the practical question of how documents get broken into the pieces retrieval actually searches over.

In one sentence

RAG combines retrieval and generation into one system, grounding a language model’s answers in real, retrieved content rather than relying purely on frozen training knowledge — a technique with a genuine, published 2020 research origin, now central to how most production AI products handle private, recent, or highly specific information.

Related Terms

• Retrieval
• Knowledge Base
• Grounding
• Hallucination
• Inference