The Evaluation Dataset article covered the test material every evaluation technique depends on. This article covers the first specific, named quality dimension that material actually gets used to measure: faithfulness.
The simple definition
Faithfulness measures whether a model’s generated answer is genuinely supported by, and consistent with, the source material it was given — specifically catching claims the answer makes that the source content doesn’t actually support. Recall from the Grounding article, back in the RAG & Retrieval phase, that a grounded answer is one traceable back to real, retrieved evidence. Faithfulness is the concrete, measurable metric that actually checks for exactly that property, turning grounding from a qualitative goal into a real, quantifiable score.
Why this metric exists as its own, separate check
Recall from the Hallucination article’s core mechanism — a language model can generate fluent, confident text that isn’t actually true, even when given real, relevant source material to work from. A response can partially use its retrieved context correctly while also quietly adding an unsupported detail the source never actually said — recall from the Grounding article’s own “partially grounded” concept. Faithfulness exists specifically to catch this exact, subtle failure mode, checking each individual claim a response makes against the actual source material, rather than judging the response as a single, undifferentiated whole.
flowchart LR
A[Retrieved source content] --> B[Model generates an answer]
B --> C[Break the answer into individual claims]
C --> D{Is each claim actually supported by the source?}
D -->|Yes| E[Faithful]
D -->|No| F[Unfaithful — a hallucinated addition]How this actually gets measured in practice
This is worth being concrete about, since it’s a real, specific calculation, not a vague, holistic judgment. Recall from the RAG Evaluation article’s introduction of RAGAS, the real, published evaluation framework.
RAGAS calculates faithfulness by first breaking a generated answer down into individual, separate factual claims, then checking each claim against the retrieved source content — typically using the LLM-as-a-judge technique covered earlier in this phase — and calculating the final score as the proportion of claims that were genuinely supported.
A response making five claims, four of them backed by the source and one unsupported, would score 0.8 — a real, precise, and genuinely useful number for tracking exactly how faithful a system’s outputs typically are.
ANALOGY vs. TECHNICAL REALITY
Analogy: Think of fact-checking a news article by going through it claim by claim, checking each individual assertion against the reporter’s actual cited sources — an article might get most of its facts right while still containing one specific, unsupported claim slipped in without proper backing, and a genuine fact-check catches that one claim specifically, rather than giving the whole article a single pass-or-fail grade.
Where this breaks down: A human fact-checker applies genuine editorial judgment about what counts as adequately supported. Faithfulness, as calculated by RAGAS and similar frameworks, is a mechanical, LLM-as-a-judge-driven calculation — inheriting the exact same position, verbosity, and self-preference biases covered in the LLM-as-a-Judge article, meaning the faithfulness score itself is a genuinely useful but imperfect proxy, not an infallible verdict.
A concrete example, layered
For a simple beginner example: asked “what’s our return policy,” a RAG system’s response saying “returns are accepted within 30 days, and refunds are typically processed within 3-5 business days” would score as unfaithful if the retrieved policy document only actually mentioned the 30-day window, with the 3-5 business day detail added from the model’s own general assumptions rather than the actual source.
For a production example: enterprise RAG deployments in regulated industries, echoing the real, cited stakes covered in the Grounding article, commonly set a minimum faithfulness threshold — flagging or blocking any response that scores below it — as a real, automated safety check before a generated answer is ever shown to a customer or used in a regulated decision.
Why a high faithfulness score doesn’t guarantee a fully correct answer
It’s worth being honest about a genuine, important limitation here, tying directly back to the Grounding article’s own caveat. A response can be perfectly faithful to its retrieved source material while still being wrong, if the source material itself was outdated or incorrect — recall from the Knowledge Base article’s principle that a retrieval system’s quality is capped by the quality of what it retrieves from. Faithfulness specifically measures whether the model stayed true to what it was given; it says nothing about whether what it was given was actually accurate in the first place.
The real story: when an unfaithful answer caused real, immediate business damage
This deserves telling in full, because it’s a genuine, very recent, and publicly acknowledged case of exactly the failure faithfulness evaluation exists to catch. In April 2025, users of Cursor, a popular AI coding assistant, began getting logged out when switching between devices. When they contacted support, an AI support bot — posing as an agent named “Sam” — confidently told them this was “expected behavior” under a new session-limit policy.
There was no such policy. The bot had invented it entirely, with no source material actually supporting the claim.
The story spread rapidly across Hacker News and Reddit, triggering a wave of subscription cancellations and public criticism from AI commentators, including Google’s former chief decision scientist Cassie Kozyrkov, who noted the deeper problem: “users hate being tricked by a machine posing as a human.” Cursor’s co-founder Michael Truell issued a public apology on Reddit, writing plainly: “We have no such policy. You’re of course free to use Cursor on multiple machines.
Unfortunately, this is an incorrect response from a front-line AI support bot.” The underlying cause, Truell explained, was a real backend session-security change that had inadvertently logged users out — the bot then fabricated a plausible-sounding policy to explain behavior it had no actual documentation for, precisely the kind of unsupported claim a real faithfulness check, run before that response ever reached a user, would have caught.
Common misconception
Four possible claim judgments
A claim does not always fit neatly into supported or unsupported:
| Judgment | Meaning |
|---|---|
| Supported | The context provides enough evidence for the claim |
| Partially supported | Only part of the claim follows from the context |
| Contradicted | The context says the opposite |
| Unverifiable | The supplied context does not settle the claim |
Example context: “The plan costs $10 per month in the United States.”
Claim: “The plan costs $10 everywhere and includes unlimited storage.”
The United States price is supported. “Everywhere” is not supported, and unlimited storage is unverifiable from this context. Breaking compound sentences into smaller claims prevents one supported phrase from hiding unsupported additions.
Citation faithfulness
A response may contain citations but still be unfaithful if a citation does not support the nearby claim. Evaluation should open the cited passage and check entailment—whether the evidence actually justifies what the answer says.
Ragas and model-based groundedness evaluators can scale this check, while human audits remain important for complicated or high-impact claims.
Calculate it claim by claim
Retrieved context says: “The museum opens at 9 a.m. and closes at 5 p.m.”
Model answer: “The museum opens at 9 a.m., closes at 5 p.m., and is free on Mondays.”
| Claim | Supported by context? |
|---|---|
| Opens at 9 a.m. | Yes |
| Closes at 5 p.m. | Yes |
| Free on Mondays | No |
faithfulness = 2 supported claims / 3 total claims = 0.67The unsupported Monday claim is a hallucination relative to the supplied context, even if the museum happens to be free on Mondays in the real world.
What the metric needs
Faithfulness compares the response with the retrieved context. It does not require a reference answer, but it does require a reliable way to break the answer into claims and decide whether each claim follows from the context.
Ragas provides faithfulness implementations that perform this claim-and-support process with an evaluator model. The score depends on the evaluator and prompts, so teams should manually review samples, especially near decision thresholds.
Faithful, correct, and relevant are separate
faithful -> supported by supplied context
correct -> agrees with verified truth/reference
relevant -> addresses the user's questionVerified sources
A frequent beginner assumption: that “faithfulness” and “correctness” mean essentially the same thing — that a faithful answer is automatically a correct one. As this article’s limitation explained, and as the next article, Answer Correctness, will make precise, these are genuinely separate, independently measured dimensions — a response can be faithful but wrong (if the source itself was wrong), or, in principle, land on a correct answer while adding unsupported detail beyond what its source actually said.
Where this fits in what comes next
You now understand the specific metric checking whether an answer sticks faithfully to its source material. The next article, Answer Correctness, covers the genuinely distinct, complementary question this article’s own limitation raised — whether an answer is actually, objectively right, regardless of how faithfully it used its sources.
In one sentence
Faithfulness measures whether each individual claim in a generated answer is genuinely supported by the retrieved source material, turning the Grounding article’s qualitative goal into a real, calculable score — commonly computed via RAGAS’s claim-by-claim LLM-as-a-judge breakdown — though a faithful answer can still be wrong if the underlying source material itself wasn’t accurate.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed