TechByteByByte

Prompt Construction for RAG

Closing Level 5: assembling the final prompt from constructed context, system instructions, and the user's question, directly connecting to the Prompt Engineering course's foundations.

#RAG#AI#Prompt Engineering#Level 5

Begin with the problem

A RAG prompt must clearly separate the user’s question, retrieved evidence, and answer rules. Otherwise the model may ignore evidence or treat document text as instructions.

user question → transform/retrieve → construct context → grounded answer + citations

What you will learn

  • Explain Prompt Construction for RAG in simple language before using its technical details.
  • Follow the mechanism step by step through a small RAG example.
  • Connect this topic to the modules before and after it.
  • Decide when to use it, when not to use it, and what to measure in production.

Current real-system grounding: Google documents returned grounding information and citations in Gemini File Search. A citation exposes a source; your application still must verify that the source supports the claim.

The product example proves that the pattern is used in a real system. It does not mean every provider uses the same hidden algorithm, defaults, limits, or pricing.

1. The problem this module solves

Module 21 produced clean, well-organized context. This module closes Level 5 by covering the final assembly step: turning that context, a system instruction, and the user’s question into the actual prompt sent to the LLM. This directly connects your prior Prompt Engineering course’s techniques to everything this course has built up to this point.

2. The Typical RAG Prompt Structure

System Instructions
   +
Retrieved Context (Module 21's constructed context)
   +
User's Question
   ↓
LLM
Example, using this course's recurring HR scenario:

SYSTEM: "You are a helpful HR assistant. Be concise and cite the
        source when possible."

CONTEXT: "[Source: travel_policy_2026] London hotel limit is
         $250/night."

QUESTION: "What's the London hotel reimbursement limit?"

3. Instructing the Model to Stay Grounded — Directly Connecting to

Module 32’s Hallucination Prevention

This is a really important, specific prompt instruction worth calling out directly:

"Answer using ONLY the information in the context above. If the
context doesn't contain the answer, say so explicitly."

This single instruction directly connects to your Generative AI course’s hallucination discussion — explicitly telling the model to stay grounded in provided context, and to acknowledge uncertainty rather than fabricate an answer, really reduces (though, as covered later in Module 26 of this course, does not eliminate) hallucination risk.

4. Context Placement — Building Directly on Module 21

Where should CONTEXT sit within the prompt -- before or after the
QUESTION?

Really, both patterns are used in practice, but placing CONTEXT
BEFORE the question is common, since it lets the model "read" the
relevant material FIRST, before being asked to reason about a
specific question against it -- similar to how a human might read a
reference document before answering a question about it.

This directly connects to Module 21’s lost-in-the-middle discussion: regardless of before/after ordering, the internal ordering of multiple context chunks (most relevant first or last) remains really important.

5. Citation Instructions — Setting Up Module 23

"Cite the source when possible" -- this single instruction, combined
with Module 9's metadata (carried through to Module 21's constructed
context, which includes [Source: ...] tags), enables the model to
REALLY reference where its answer came from.

Without metadata explicitly included IN the context (Module 21), no prompt instruction alone could make citation possible — the model can only cite what it can actually see in its input. This is a direct, concrete example of how Modules 9, 21, and this module’s prompt construction work together as one coherent pipeline.

6. A Real Developer Example — The Complete Assembly

TechCorp's HR assistant, assembling a FULL prompt from everything
this course has built:

1. Module 19's TRANSFORMED query retrieves relevant chunks
2. Module 18's RERANKING refines their order
3. Module 21's CONTEXT CONSTRUCTION deduplicates, filters, and
   orders them (most relevant first)
4. THIS module assembles the final prompt:
   - SYSTEM instructions (role, tone, citation requirement)
   - The constructed CONTEXT (with source metadata intact)
   - The user's ORIGINAL question (not the transformed search
     query -- the model should answer the user's ACTUAL question,
     even though a DIFFERENT, transformed query was used for
     retrieval)
   - An explicit GROUNDING instruction (Section 3)

Every single module from Level 3 through Level 5 contributes
DIRECTLY to this one final, assembled prompt.

7. A Simple Agentic AI Connection

An agent constructing its own prompts for each reasoning step follows this exact same pattern — system instructions defining its role and constraints, relevant retrieved or accumulated context, and the specific sub-task or question at hand, assembled deliberately at every step of its multi-step loop (your Generative AI course’s Module 29), not just once at the very beginning of a task.

8. How Is This Used in AI?

🤖 How Is This Used in AI?

This prompt assembly pattern — system instructions, constructed context, grounding instructions, user question — is the standard structure underlying essentially every production RAG application’s final generation call, directly built from your Prompt Engineering course’s techniques applied specifically to retrieval-augmented generation.

9. Real-World Applications

  • Every RAG application’s final generation step uses some version of this prompt structure
  • Customer-facing chatbots requiring both grounded accuracy and real source citation
  • Internal knowledge assistants balancing helpfulness with strict factual grounding

10. Common Mistakes

Incorrect idea: Forgetting an explicit grounding instruction.

Why it is incorrect: As shown directly in Section 3, this directly connects to hallucination mitigation — omitting it removes a real, low-cost safeguard.

Incorrect idea: Replacing the user’s original question with the transformed search query in the final prompt.

Why it is incorrect: As shown directly in Section 6, the model should answer the user’s ACTUAL question — the transformed query (Module 19) is for RETRIEVAL only, not for what the model is ultimately asked to answer.

Incorrect idea: Requesting citations without including source metadata in the context itself.

Why it is incorrect: As shown directly in Section 5, the model can only cite what it can really see — this requires Module 9 and 21’s metadata pipeline to already be in place.

11. Limitations

  • Even a well-constructed grounding prompt doesn’t provide a complete guarantee against hallucination (Module 26 covers this directly) — it’s a real, valuable mitigation, not an absolute solution
  • Prompt structure and instructions really benefit from evaluation (Module 32) against real queries, not just intuition about what “should” work

12. Quick Reference — The Whole Idea in One Diagram

SYSTEM instructions (role, tone, citation requirement)
   +
CONTEXT (Module 21's clean, ordered, metadata-rich construction)
   +
GROUNDING instruction (Section 3 -- "answer ONLY from context")
   +
USER's original question (not the transformed search query, Module
19)
   ↓
LLM -> grounded, cited answer

13. Code — Assembling the Complete RAG Prompt

🎯 Target of this example: implement Section 6’s complete assembly example directly — combining system instructions, Module 21’s constructed context, a grounding instruction, and the user’s original question into one final, well-structured prompt, then sending it to the model.

Example 1 — Simple

def build_rag_prompt(system_instructions: str, context: str, question: str) -> dict:
    """Assembles the final RAG prompt from THREE distinct parts,
    exactly Section 2's structure, WITH Section 3's grounding
    instruction included explicitly."""
    user_message = (
        f"Context:\n{context}\n\n"
        f"Question: {question}\n\n"
        f"Answer using ONLY the information in the context above. "
        f"If the context doesn't contain the answer, say so explicitly."
    )
    return {"system": system_instructions, "user_message": user_message}

system_instructions = "You are a helpful HR assistant. Be concise and cite the source when possible."
context = "[Source: travel_policy_2026]\nLondon hotel limit is $250/night."
question = "What's the London hotel reimbursement limit?"

prompt = build_rag_prompt(system_instructions, context, question)
print("SYSTEM:", prompt["system"])
print("\nUSER MESSAGE:")
print(prompt["user_message"])

Expected Output:

SYSTEM: You are a helpful HR assistant. Be concise and cite the
source when possible.

USER MESSAGE:
Context:
[Source: travel_policy_2026]
London hotel limit is $250/night.

Question: What's the London hotel reimbursement limit?

Answer using ONLY the information in the context above. If the
context doesn't contain the answer, say so explicitly.

What we conclude from this example: the assembled prompt cleanly separates system role instructions from the user-facing message containing context, the question, and an explicit grounding instruction — exactly Section 2 and 3’s structure, ready to be sent directly to an LLM.

Example 2 — Intermediate

import anthropic

client = anthropic.Anthropic()

def build_rag_prompt(system_instructions: str, context: str, question: str) -> dict:
    user_message = (
        f"Context:\n{context}\n\n"
        f"Question: {question}\n\n"
        f"Answer using ONLY the information in the context above. "
        f"If the context doesn't contain the answer, say so explicitly."
    )
    return {"system": system_instructions, "user_message": user_message}

def generate_rag_answer(system_instructions: str, context: str, question: str) -> str:
    """Sends the assembled prompt to the LLM -- the FINAL step of
    the entire RAG pipeline this course has built, Level 3 through
    Level 5."""
    prompt = build_rag_prompt(system_instructions, context, question)
    response = client.messages.create(
        model="claude-sonnet-4-6", max_tokens=100,
        system=prompt["system"],
        messages=[{"role": "user", "content": prompt["user_message"]}]
    )
    return response.content[0].text

context = "[Source: travel_policy_2026]\nLondon hotel limit is $250/night. Standard limit is $200/night."
answer = generate_rag_answer(
    "You are a helpful HR assistant. Be concise and cite the source when possible.",
    context, "What's the London hotel reimbursement limit?",
)
print(answer)

Expected Output:

The London hotel reimbursement limit is $250 per night, according to
the travel policy 2026 document.

What we conclude from this example: the model correctly answers using only the provided context and includes a source reference — a direct, working demonstration of the full prompt construction pipeline producing a really grounded, cited answer.

Example 3 — Production Grade

import anthropic
from dataclasses import dataclass

client = anthropic.Anthropic()

@dataclass
class RAGPromptResult:
    system_instructions: str
    assembled_context: str
    original_question: str
    answer: str
    grounding_instruction_included: bool

class RAGPromptBuilder:
    """A production-style prompt builder making Section 3's grounding
    instruction and Section 5's citation setup STRUCTURAL parts of
    every prompt, rather than something a developer could forget to
    include on a case-by-case basis."""

    DEFAULT_GROUNDING_INSTRUCTION = (
        "Answer using ONLY the information in the context above. "
        "If the context doesn't contain the answer, say so explicitly."
    )

    def __init__(self, system_instructions: str):
        self.system_instructions = system_instructions

    def build_and_generate(self, context: str, original_question: str) -> RAGPromptResult:
        user_message = (
            f"Context:\n{context}\n\n"
            f"Question: {original_question}\n\n"
            f"{self.DEFAULT_GROUNDING_INSTRUCTION}"
        )

        response = client.messages.create(
            model="claude-sonnet-4-6", max_tokens=100,
            system=self.system_instructions,
            messages=[{"role": "user", "content": user_message}]
        )

        return RAGPromptResult(
            system_instructions=self.system_instructions, assembled_context=context,
            original_question=original_question, answer=response.content[0].text,
            grounding_instruction_included=True,  # ALWAYS true -- structurally guaranteed
        )

builder = RAGPromptBuilder("You are a helpful HR assistant. Be concise and cite the source when possible.")

context = "[Source: travel_policy_2026]\nLondon hotel limit is $250/night. Standard limit is $200/night."
result = builder.build_and_generate(context, "What's the London hotel reimbursement limit?")

print(f"Question: {result.original_question}")
print(f"Answer: {result.answer}")
print(f"Grounding instruction included: {result.grounding_instruction_included}")

Expected Output:

Question: What's the London hotel reimbursement limit?
Answer: The London hotel reimbursement limit is $250 per night,
according to the travel policy 2026 document.
Grounding instruction included: True

What we conclude from this example: the RAGPromptBuilder class structurally guarantees the grounding instruction is ALWAYS included in every generated prompt — grounding_instruction_included is always True by design, not something that could accidentally be omitted by a developer forgetting to add it manually, directly implementing Section 10’s warning as an architectural safeguard rather than a convention to remember.

14. Interview Questions

Q: What are the standard components of a well-constructed RAG prompt?

Ans: A typical RAG prompt combines system instructions (defining the model’s role, tone, and behavioral requirements like citation), the constructed context from retrieval (Module 21’s filtered, deduplicated, and ordered chunks, ideally with source metadata included), the user’s original question, and an explicit grounding instruction telling the model to answer only from the provided context and to acknowledge when the context doesn’t contain the answer.

Q: Why is it important to use the user’s original question in the final prompt, rather than the transformed search query from Module 19?

Ans: The transformed query (rewritten for effective retrieval) exists specifically to improve search results — it’s not necessarily what the user actually asked, and might use different, more formal phrasing. The model should be answering the user’s real, original question, using the context that transformed query happened to retrieve — using the transformed query as the final question instead could produce an answer that technically responds to the search phrasing but doesn’t directly address what the user actually wanted to know.

Q: Why does the grounding instruction (“answer only using the provided context”) really help reduce hallucination, even though it’s just a prompt instruction rather than a technical constraint?

Ans: This instruction explicitly directs the model toward calibrated, grounded behavior — encouraging it to acknowledge when information is really missing rather than fabricating a plausible-sounding answer from its general training knowledge. While this doesn’t provide an absolute technical guarantee against hallucination, it’s a real, low-cost mitigation that measurably shifts model behavior toward staying grounded in the actual provided context, directly connecting to the broader hallucination mitigation strategies covered later in this course.

Q: Why would a production system implement prompt construction as a reusable class with a hard-coded grounding instruction, rather than letting each call site write its own custom prompt?

Ans: Making the grounding instruction structurally guaranteed — always included, not optional — prevents a real, easy mistake: a developer forgetting to add it on a specific call, silently losing an important hallucination mitigation for that particular request. A reusable builder class also ensures consistency across an application’s different RAG features, making prompt structure changes (like updating the grounding instruction’s wording) a single, centralized update rather than requiring changes scattered across many different code locations.

15. What You Should Remember

  • A well-constructed RAG prompt combines system instructions, constructed context (Module 21), a grounding instruction, and the user’s original question — verified directly through a working end-to-end pipeline producing a really grounded, cited answer.
  • The grounding instruction directly connects to hallucination mitigation — a real, low-cost safeguard, though not a complete guarantee.
  • Structural enforcement of prompt components (like a builder class guaranteeing the grounding instruction is always present) prevents real, easy-to-make omission mistakes — verified directly through a production-style class making this guarantee explicit.

16. Quick Practice

Write out a complete RAG prompt (system instructions, context, grounding instruction, question) for a customer support scenario of your choosing — identify which specific words in your grounding instruction are doing the actual work of reducing hallucination risk.

17. Next Step

Next: Module 23 — Grounded Generation & Citations — Level 6 begins here: what it actually means for an answer to be grounded, and how to attach real, verifiable citations to generated responses.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed