RAG Evaluation – RAG | TechByteByByte

Begin with the problem

A polished answer can hide poor retrieval, and perfect retrieval can still lead to a poor answer. RAG evaluation measures both layers separately and together.

evaluation set + production traces → metrics → diagnosis → safer improvement

What you will learn

Explain RAG Evaluation in simple language before using its technical details.
Follow the mechanism step by step through a small RAG example.
Connect this topic to the modules before and after it.
Decide when to use it, when not to use it, and what to measure in production.

Current real-system grounding: OpenAI’s evaluation guide and Google’s File Search documentation ground the production practices discussed here. Limits, costs, and supported models change, so verify them before deployment.

The product example proves that the pattern is used in a real system. It does not mean every provider uses the same hidden algorithm, defaults, limits, or pricing.

1. The problem this module solves

Every module in this course has referenced evaluation as the way to really verify a decision, rather than assuming it worked. This module gives evaluation its full, dedicated treatment — building directly on Module 24’s diagnostic principle: evaluate EACH pipeline stage separately, not just the final answer.

2. The Core Principle — Evaluate Every Layer, Not Just the Final

Answer

Documents -> Chunks -> Retrieval -> Ranking -> Context -> Generation
-> Answer

Directly extending Module 24’s diagnostic principle: since failures can really occur at ANY stage, evaluation needs to measure EACH stage separately. Evaluating only the final answer tells you THAT something is wrong, but not WHERE — exactly the same problem Module 24’s diagnostic process was built to solve.

3. Retrieval Metrics — Precision and Recall, Refreshed

Precision@K:      of the TOP K retrieved results, what FRACTION are
                 REALLY relevant?

Recall@K:            of ALL really relevant documents that EXIST,
                    what FRACTION were found in the top K retrieved
                    results?

Worked example:

5 documents were retrieved. 4 documents are REALLY relevant
(exist somewhere in the knowledge base).

Retrieved: [doc_5, doc_2, doc_9, doc_1, doc_7]
Really relevant: {doc_2, doc_1, doc_3, doc_8}

Precision@3 (top 3: doc_5, doc_2, doc_9):      1 of 3 are relevant
                                              (doc_2) -> 0.33

Recall@3:                                            1 of 4 REAL
                                                    relevant docs
                                                    found -> 0.25

The real trade-off these two metrics expose: you could achieve perfect recall by retrieving EVERYTHING (guaranteed to include every relevant document) — but precision would collapse. You could achieve perfect precision by retrieving only ONE, extremely-confident result — but recall would likely suffer. Real evaluation tracks BOTH, together.

4. Generation Metrics — Beyond Simple Correctness

FAITHFULNESS:      does the generated answer REALLY reflect the
                  retrieved context? (directly Module 23's
                  groundedness check, formalized as a metric)

RELEVANCE:            does the answer actually ADDRESS the question
                     that was asked?

COMPLETENESS:            does the answer cover everything the
                        retrieved context really supports, or
                        does it MISS relevant details that WERE
                        available?

These require really more nuanced judgment than retrieval’s precision/recall — often implemented via LLM-as-judge (a separate model call evaluating the generated answer), directly connecting to Module 23’s groundedness verification pattern.

5. The Complete End-to-End Evaluation Framework

For each TEST QUESTION in a golden dataset:

1. Question
2. EXPECTED evidence (which chunk(s) SHOULD be retrieved)
3. EXPECTED answer (what a GOOD answer looks like)

Then evaluate:

1. Did we retrieve the CORRECT evidence? (Precision/Recall@K,
   Section 3)
2. Did generation actually USE that evidence? (Faithfulness,
   Section 4)
3. Is the answer really CORRECT? (comparison against expected
   answer)
4. Is the answer really GROUNDED? (Module 23)
5. Are CITATIONS correct? (Module 23's citation-correctness caveat)

This is really the core evaluation framework this entire course has been building toward — every prior module’s quality concern gets a measurable, trackable home here.

6. A Real Developer Example

TechCorp builds a GOLDEN DATASET of 50 representative real employee
questions, each with:

- The EXPECTED chunk (e.g., the London hotel exception paragraph)
- The EXPECTED answer ("$250 per night")

Before shipping a change (a new chunking strategy, Module 8; a
different embedding model, Module 10; a reranking tweak, Module 18):

1. Run the CURRENT system against all 50 questions -> BASELINE
   scores (precision, recall, faithfulness, correctness)
2. Make the proposed change
3. Run the UPDATED system against the SAME 50 questions -> COMPARISON
   scores
4. Compare: did retrieval precision/recall really IMPROVE? Did
   faithfulness stay high? Did correctness improve or REGRESS?

This is EXACTLY the systematic, repeatable evaluation that prevents
shipping a change that "felt" better in casual testing but actually
degraded performance on cases not manually checked.

7. A Simple Agentic AI Connection

An agent’s retrieval tool calls (Module 29 of this course) benefit directly from this same per-stage evaluation approach — measuring whether the agent’s chosen search queries really retrieved relevant information, separately from whether its final synthesized answer was correct, allows diagnosing whether an agent’s poor performance stems from bad search strategy versus bad reasoning over otherwise-good results.

8. How Is This Used in AI?

🤖 How Is This Used in AI?

Mature RAG engineering teams maintain real, ongoing evaluation infrastructure — golden datasets, automated per-stage scoring, and regular re-evaluation whenever any pipeline component changes — precisely because “it worked when I tried it” is really insufficient confidence for a system serving real users, exactly mirroring your Generative AI course’s evaluation module applied specifically to RAG’s multi-stage pipeline.

9. Real-World Applications

Regression testing before deploying changes to any pipeline component (chunking, embedding model, reranking, prompts)
Comparing candidate embedding models or vector database configurations against real, representative data
Ongoing production quality monitoring across really distinct metrics

10. Common Mistakes

Incorrect idea: Evaluating only the final answer’s correctness.

Why it is incorrect: As shown directly in Section 2, this tells you THAT something failed, but not WHERE — Module 24’s diagnostic value is lost entirely.

Incorrect idea: Tracking only precision or only recall, not both.

Why it is incorrect: As shown directly in Section 3, these represent a real trade-off — tracking just one gives an incomplete, potentially misleading picture.

Incorrect idea: Relying purely on casual, manual testing before deploying changes.

Why it is incorrect: As shown directly in Section 6, this can miss real regressions on cases that weren’t manually checked.

11. Limitations

Building and maintaining a really representative golden dataset requires real, ongoing effort — a stale or narrow dataset provides false confidence
LLM-as-judge evaluation for generation metrics (Section 4) has its own real limitations — the judge itself can be imperfect, directly connecting back to Module 25’s hallucination discussion applied to the EVALUATOR itself

12. Quick Reference — The Whole Idea in One Diagram

EVALUATE EVERY STAGE, not just the final answer:

RETRIEVAL:      Precision@K, Recall@K (Section 3's real trade-off)

GENERATION:        Faithfulness, Relevance, Completeness (Section 4)

END-TO-END:            golden dataset with expected evidence AND
                      expected answers -> BASELINE before changes,
                      COMPARE after (Section 6)

13. Code — Implementing Precision@K, Recall@K, and End-to-End

Evaluation

🎯 Target of this example: implement Section 3’s worked example directly, then build toward Section 6’s complete golden-dataset comparison workflow, verifying a proposed change really improves (or regresses) system quality.

Example 1 — Simple

def precision_at_k(retrieved_ids: list, relevant_ids: set, k: int) -> float:
    """Directly implements Section 3's Precision@K definition."""
    top_k = retrieved_ids[:k]
    relevant_in_top_k = sum(1 for doc_id in top_k if doc_id in relevant_ids)
    return relevant_in_top_k / k

def recall_at_k(retrieved_ids: list, relevant_ids: set, k: int) -> float:
    """Directly implements Section 3's Recall@K definition."""
    top_k = retrieved_ids[:k]
    relevant_in_top_k = sum(1 for doc_id in top_k if doc_id in relevant_ids)
    return relevant_in_top_k / len(relevant_ids)

# Section 3's exact worked example
retrieved = ["doc_5", "doc_2", "doc_9", "doc_1", "doc_7"]
truly_relevant = {"doc_2", "doc_1", "doc_3", "doc_8"}

p_at_3 = precision_at_k(retrieved, truly_relevant, k=3)
r_at_3 = recall_at_k(retrieved, truly_relevant, k=3)
p_at_5 = precision_at_k(retrieved, truly_relevant, k=5)
r_at_5 = recall_at_k(retrieved, truly_relevant, k=5)

print(f"Precision@3: {p_at_3:.2f}, Recall@3: {r_at_3:.2f}")
print(f"Precision@5: {p_at_5:.2f}, Recall@5: {r_at_5:.2f}")

Expected Output:

Precision@3: 0.33, Recall@3: 0.25
Precision@5: 0.40, Recall@5: 0.50

What we conclude from this example: as K increases from 3 to 5, both precision and recall really improve here (more relevant documents are captured) — but notice precision stays relatively low (0.40 at best) since the retrieval system still returns a meaningful number of irrelevant documents, exactly Section 3’s trade-off made directly measurable.

Example 2 — Intermediate

def precision_at_k(retrieved_ids: list, relevant_ids: set, k: int) -> float:
    top_k = retrieved_ids[:k]
    return sum(1 for doc_id in top_k if doc_id in relevant_ids) / k

def recall_at_k(retrieved_ids: list, relevant_ids: set, k: int) -> float:
    top_k = retrieved_ids[:k]
    return sum(1 for doc_id in top_k if doc_id in relevant_ids) / len(relevant_ids)

def evaluate_golden_dataset(test_cases: list, retrieval_fn, k: int = 5) -> dict:
    """Runs Precision@K and Recall@K across an ENTIRE golden dataset
    (Section 6's core evaluation practice), reporting AVERAGE
    performance rather than a single test case's result."""
    precisions, recalls = [], []
    for case in test_cases:
        retrieved = retrieval_fn(case["question"])
        precisions.append(precision_at_k(retrieved, case["relevant_ids"], k))
        recalls.append(recall_at_k(retrieved, case["relevant_ids"], k))

    return {"avg_precision": round(sum(precisions) / len(precisions), 3),
            "avg_recall": round(sum(recalls) / len(recalls), 3)}

def mock_retrieval_v1(question: str) -> list:
    """Simulates a BASELINE retrieval system."""
    mock_results = {
        "London hotel limit?": ["doc_5", "doc_2", "doc_9", "doc_1", "doc_7"],
        "Domestic hotel limit?": ["doc_9", "doc_1", "doc_6", "doc_3", "doc_2"],
    }
    return mock_results[question]

test_cases = [
    {"question": "London hotel limit?", "relevant_ids": {"doc_2", "doc_1", "doc_3", "doc_8"}},
    {"question": "Domestic hotel limit?", "relevant_ids": {"doc_1", "doc_3"}},
]

results = evaluate_golden_dataset(test_cases, mock_retrieval_v1, k=5)
print(f"Baseline system -- Avg Precision@5: {results['avg_precision']}, Avg Recall@5: {results['avg_recall']}")

Expected Output:

Baseline system -- Avg Precision@5: 0.4, Avg Recall@5: 0.75

What we conclude from this example: aggregating precision and recall across MULTIPLE test questions (rather than just one, as in Example 1) gives a real, representative measure of overall system quality — exactly Section 6’s golden dataset evaluation practice, producing a baseline score ready to compare against after any proposed system change.

Example 3 — Production Grade

from dataclasses import dataclass

@dataclass
class EvaluationComparison:
    baseline_precision: float
    updated_precision: float
    baseline_recall: float
    updated_recall: float
    precision_improved: bool
    recall_improved: bool

def precision_at_k(retrieved_ids: list, relevant_ids: set, k: int) -> float:
    top_k = retrieved_ids[:k]
    return sum(1 for doc_id in top_k if doc_id in relevant_ids) / k

def recall_at_k(retrieved_ids: list, relevant_ids: set, k: int) -> float:
    top_k = retrieved_ids[:k]
    return sum(1 for doc_id in top_k if doc_id in relevant_ids) / len(relevant_ids)

def evaluate_golden_dataset(test_cases: list, retrieval_fn, k: int = 5) -> dict:
    precisions, recalls = [], []
    for case in test_cases:
        retrieved = retrieval_fn(case["question"])
        precisions.append(precision_at_k(retrieved, case["relevant_ids"], k))
        recalls.append(recall_at_k(retrieved, case["relevant_ids"], k))
    return {"avg_precision": round(sum(precisions) / len(precisions), 3),
            "avg_recall": round(sum(recalls) / len(recalls), 3)}

def compare_before_after_change(test_cases: list, baseline_fn, updated_fn, k: int = 5) -> EvaluationComparison:
    """The FULL production workflow from Section 6 -- run BOTH the
    baseline and updated system against the SAME golden dataset, and
    explicitly determine whether the proposed change really
    improved things, BEFORE shipping it."""
    baseline_results = evaluate_golden_dataset(test_cases, baseline_fn, k)
    updated_results = evaluate_golden_dataset(test_cases, updated_fn, k)

    return EvaluationComparison(
        baseline_precision=baseline_results["avg_precision"],
        updated_precision=updated_results["avg_precision"],
        baseline_recall=baseline_results["avg_recall"],
        updated_recall=updated_results["avg_recall"],
        precision_improved=updated_results["avg_precision"] >= baseline_results["avg_precision"],
        recall_improved=updated_results["avg_recall"] >= baseline_results["avg_recall"],
    )

def mock_retrieval_v1(question: str) -> list:
    mock_results = {
        "London hotel limit?": ["doc_5", "doc_2", "doc_9", "doc_1", "doc_7"],
        "Domestic hotel limit?": ["doc_9", "doc_1", "doc_6", "doc_3", "doc_2"],
    }
    return mock_results[question]

def mock_retrieval_v2_improved(question: str) -> list:
    """Simulates an IMPROVED system (e.g., after adding reranking,
    Module 18) -- really relevant docs now rank higher."""
    mock_results = {
        "London hotel limit?": ["doc_2", "doc_1", "doc_3", "doc_5", "doc_9"],
        "Domestic hotel limit?": ["doc_1", "doc_3", "doc_9", "doc_6", "doc_2"],
    }
    return mock_results[question]

test_cases = [
    {"question": "London hotel limit?", "relevant_ids": {"doc_2", "doc_1", "doc_3", "doc_8"}},
    {"question": "Domestic hotel limit?", "relevant_ids": {"doc_1", "doc_3"}},
]

comparison = compare_before_after_change(test_cases, mock_retrieval_v1, mock_retrieval_v2_improved, k=5)

print(f"Precision: {comparison.baseline_precision} -> {comparison.updated_precision} "
      f"({'IMPROVED' if comparison.precision_improved else 'REGRESSED'})")
print(f"Recall: {comparison.baseline_recall} -> {comparison.updated_recall} "
      f"({'IMPROVED' if comparison.recall_improved else 'REGRESSED'})")

Expected Output:

Precision: 0.4 -> 0.5 (IMPROVED)
Recall: 0.75 -> 0.875 (IMPROVED)

What we conclude from this example: the comparison correctly shows BOTH precision and recall really improving after the simulated reranking change — the reordered results (v2) place more really relevant documents within the top-5 than v1 did. This explicit, measured comparison — rather than assuming a reranking change helped based on intuition alone — is exactly the discipline Section 6 described: run both versions against the same golden dataset, and let the numbers, not assumptions, determine whether a proposed change is really worth shipping.

14. Interview Questions

Q: Why is it insufficient to evaluate only the final generated answer’s correctness in a RAG system?

Ans: Evaluating only the final answer tells you THAT something is wrong when a test case fails, but not WHERE in the pipeline the actual problem occurred — directly connecting to Module 24’s diagnostic principle. A failure could stem from retrieval never finding the right chunk, ranking placing it too low, context construction filtering it out, or generation failing to use it correctly despite it being present. Evaluating each pipeline stage separately (retrieval quality, generation faithfulness, final correctness) is necessary to actually diagnose and fix the real root cause.

Q: Explain Precision@K and Recall@K, and describe the real trade-off between them.

Ans: Precision@K measures what fraction of the top K retrieved results are really relevant — a measure of how much of what you retrieved is actually useful. Recall@K measures what fraction of all really relevant documents that exist were actually found within the top K results — a measure of how complete your retrieval was. These trade off against each other: retrieving more results can improve recall (more chances to include every relevant document) but tends to reduce precision (more irrelevant results mixed in), so real evaluation tracks both together rather than optimizing for just one.

Q: What does “faithfulness” measure as a generation metric, and how does it relate to Module 23’s groundedness verification?

Ans: Faithfulness measures whether a generated answer really reflects the retrieved context it was given, rather than fabricating or extrapolating beyond it. This is directly the same underlying concept as Module 23’s groundedness verification, formalized here as a trackable evaluation metric that can be measured systematically across a golden dataset, rather than checked ad hoc for individual responses.

Q: Describe the complete workflow for evaluating whether a proposed change to a RAG system (like a new embedding model or reranking strategy) really improves quality before deploying it.

Ans: First, establish a golden dataset of representative test questions, each with expected relevant evidence and an expected answer. Run the current, baseline system against this dataset to get baseline scores across retrieval metrics (precision, recall) and generation metrics (faithfulness, correctness). Implement the proposed change, then run the updated system against the exact same golden dataset to get comparison scores. Explicitly compare baseline versus updated results across every metric to determine whether the change really improved performance, rather than assuming it helped based on casual, limited manual testing.

15. What You Should Remember

Evaluate every pipeline stage separately, not just the final answer — directly extending Module 24’s diagnostic principle to systematic, ongoing measurement.
Precision@K and Recall@K represent a real trade-off — verified directly by observing both metrics change together as K increases.
Explicit before/after comparison against a golden dataset — verified directly through a working comparison function — is the discipline that prevents shipping changes that “felt” better without actual, measured evidence.

16. Quick Practice

Design a small golden dataset (3-4 test cases) for evaluating a RAG system in a domain of your choosing — specify the expected relevant document(s) and expected answer for each, exactly as this module’s framework requires.

17. Next Step

Next: Module 33 — Production RAG — the final module of this course: observability, cost, latency, caching, architecture patterns, the complete decision framework, common mistakes, case studies, and the final, unified mental model.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026