Start with the real problem
Evaluation means testing a prompt on many planned examples and measuring whether its answers meet clear rules.
A prompt that succeeds once has demonstrated one success, not reliability. Evaluation asks whether it works across the normal, difficult, and dangerous inputs the product will actually receive.
prompt + test collection → outputs → graders and humans → metrics → decisionWhat you will learn
- Create varied, realistic test cases.
- Define pass criteria before testing.
- Combine automated and human evaluation.
- Compare quality, response time, and cost across versions.
How this connects to current AI systems
Major model providers support eval-oriented workflows, but teams still need domain-specific datasets, scoring systems that have been tested against human judgments, and human review for subjective or high-stakes tasks.
1. Why This Module Exists
Module 14 introduced iteration — testing, diagnosing, refining. Module 20 formalizes the “testing” part properly. A prompt that produces a great result on the one example you tried tells you almost nothing about how it’ll perform across the hundreds or thousands of different, real inputs a production system will actually see.
2. The Idea, in Plain Language
Prompt evaluation means systematically checking how a prompt performs across many different inputs — not just trusting one good result.
Prompt
↓
Input Dataset (many varied, realistic examples)
↓
LLM
↓
Outputs
↓
Evaluation (compare against expected results / quality criteria)
↓
Metrics
↓
Prompt ImprovementThis directly extends Module 14’s iteration loop into something repeatable and measurable, rather than relying on manual spot-checking.
3. Why “It Worked Once” Isn’t Enough
Consider a prompt for extracting order numbers (Module 14’s running example). It might work perfectly on the one message you tested it with — but real customer messages vary enormously:
"my order #4471 never showed up" → works
"ORDER: 4471" → different format, might fail
"i think it was order 4471 or maybe 4417" → ambiguous, might fail
"no order number, just wondering about shipping" → correctly returns
NOT_FOUND? or guesses?A single successful test tells you the prompt can work — it tells you nothing about how often it does work across the actual range of inputs your application will encounter.
💡 The pattern to notice: “it worked” and “it’s reliable” are really different claims, and only the second one matters for a real production system.
4. Building a Test Dataset
A test dataset is simply a collection of realistic example inputs, ideally paired with the expected correct output for each one.
[
{"input": "my order #4471 never showed up", "expected": "4471"},
{"input": "ORDER: 4471", "expected": "4471"},
{"input": "no order number, just wondering about shipping", "expected": "NOT_FOUND"},
{"input": "order 8823, also is 4471 still active?", "expected": "8823"}, ← ambiguous case, worth including
]Good test datasets deliberately include:
- Typical, easy cases — the common, expected pattern
- Edge cases — missing information, unusual formatting
- Really ambiguous cases — ones where even a human might disagree on the “correct” answer, which are worth knowing about even if there’s no single clean expected output
5. Quality Criteria — What “Good” Actually Means
Depending on the task, “good” can mean different things:
Accuracy: did it produce the correct answer? (extraction,
classification tasks)
Consistency: does it produce similarly-shaped, similarly-
reliable output across different but similar inputs?
Format compliance: does it follow the exact structure requested
(Module 8), every time?
Cost: how many tokens does it use per request?
(Module 25)
Latency: how long does it take to respond?A prompt can be accurate but inconsistent in format, or fast but unreliable — “good” really depends on which of these criteria matter most for your specific application.
6. A Real Example From a Developer’s Perspective
Here’s what a real evaluation pass looks like for the order-number extraction prompt from Module 14:
Test dataset: 20 realistic customer messages, with expected outputs
Run the current prompt against all 20.
Results:
18/20 correct (90% accuracy)
2 failures:
- "order# 4471" (no space after #) → model returned "# 4471"
instead of "4471" -- a formatting edge case not covered
- "my previous order, 4471, and current one 8823" → model
returned "4471" instead of the CURRENT order "8823" -- a
genuine ambiguity the prompt doesn't address
Action: refine the prompt to handle the "#" formatting variation, and
add explicit guidance for messages mentioning multiple order numbers
(e.g., "if multiple order numbers are mentioned, extract the one most
clearly associated with the CURRENT issue").Notice: this is exactly Module 14’s iteration process, but now applied systematically across 20 cases instead of one — the two specific failures point directly at two specific, fixable gaps, rather than a vague “something’s wrong sometimes.”
7. A Simple Agentic AI Example
Evaluating an agent’s behavior is really more complex than evaluating a single response, since there are multiple points where things could go wrong — but the same core principle applies:
Test scenario: "Book me a flight to Chicago next Tuesday."
Check:
- Did the agent search BEFORE attempting to book? (rule compliance,
Module 19)
- Did it present options rather than booking automatically?
- If no flights were found, did it say so clearly rather than
fabricating a result?
Run this same scenario, and several variations of it, repeatedly --
checking not just the FINAL response, but whether the agent followed
its stated rules at every step along the way.Agent evaluation often needs to check the entire process (did it follow the rules?), not just whether the final output looked reasonable — a really important distinction from evaluating a single-response prompt.
8. How Is This Used in AI?
🤖 How Is This Used in AI?
Every serious production AI feature has some form of evaluation pipeline — running the prompt against a curated test set, tracking accuracy or other quality metrics over time, and catching regressions before a prompt change ships. This is treated as genuine engineering practice, directly analogous to automated testing in traditional software development, and it’s the practice that separates a demo that happened to work once from a system that’s actually reliable.
9. When Should You Build a Real Evaluation Process?
- Any prompt going into repeated, real production use
- Any time you’re iterating on a prompt and need to know whether a change actually helped, rather than just feeling like it might have
- Any time reliability really matters — the more consequential the task, the more this matters
10. When Is Informal Testing Enough?
- One-off, personal, low-stakes use — building a full test dataset for a single casual request is unnecessary overhead
11. Common Mistakes
Incorrect idea
Testing with only easy, typical cases.
Why it is incorrect
A test dataset that only includes the simplest, most expected inputs won’t reveal the edge cases and ambiguities that actually cause real-world failures.
Incorrect idea
Treating “it passed my test dataset” as a permanent guarantee.
Why it is incorrect
Real-world input patterns can shift over time — evaluation should be an ongoing practice, not a one-time checkbox (Module 21 covers versioning and regression testing directly).
Incorrect idea
Only checking the final output, for agents.
Why it is incorrect
As shown directly, agent evaluation often needs to verify the process (did it follow its rules?), not just whether the final result looked acceptable.
Incorrect idea
Not defining “correct” clearly before testing.
Why it is incorrect
For really ambiguous or subjective tasks, it’s worth deciding in advance what counts as an acceptable answer, rather than judging results inconsistently after the fact.
12. Limitations
- Building and maintaining a good test dataset takes real, ongoing effort — it’s not a one-time task, especially as real-world usage patterns evolve
- Evaluation tells you how a prompt performs on your test cases — it can’t guarantee performance on really novel inputs never represented in that test set
- For subjective or creative tasks, defining clear, checkable “correct” criteria is itself a real challenge, not just a formality
[Preserve] Analogy: The DMV Driving Road Test Think of evaluating a prompt like certifying a new driver at the DMV:
- The Single Drive Test (Casual spot-check): You sit in the passenger seat. The student drives 2 blocks in a straight line, parks successfully, and you say: “Great! You are a fully qualified driver.”
- This tells you they can drive under perfect conditions. It tells you nothing about how they handle parallel parking, heavy rain, roundabouts, or highway speeds.
- The DMV Driving Test Suite (Systematic Evaluation): The DMV maintains a standard test route containing:
- 3 right turns, 2 left turns across traffic, a parallel parking segment, and a highway merge.
- Every driver must run the identical route. You track exact scores: did they hit the curb? Did they yield?
- In prompt engineering, your test dataset is the standard DMV route. You measure how the prompt handles a diverse test suite, not just one lucky straight-line drive.
📊 Visual Flowchart: The Evaluation Pipeline
Here is how prompts are systematically verified against standard test sets:
graph TD
classDef test fill:#3498db,stroke:#333,stroke-width:1px,color:#fff;
classDef eval fill:#e67e22,stroke:#333,stroke-width:1px,color:#fff;
classDef metrics fill:#2ecc71,stroke:#333,stroke-width:1px,color:#fff;
subgraph TestSet ["Standard Test Set (CSV / JSON)"]
TC1["Case 1: User message + Expected output 1"]
TC2["Case 2: User message + Expected output 2"]
TC3["Case 3: User message + Expected output 3"]
end
PromptTemplate["Active Prompt Template"] --> Runner["Execution Engine:<br>Run prompt against all test cases"]:::test
TestSet --> Runner
Runner --> RawOutputs["Raw LLM Completions"]
RawOutputs --> Evaluator["Programmatic Evaluator:<br>(Check Regex, JSON schema, or LLM-as-a-judge)"]:::eval
Evaluator --> Tally["Tally Pass/Fail scores per criteria"]:::eval
Tally --> Summary["Metrics Dashboard:<br>- Accuracy: 94%<br>- Format compliance: 100%<br>- Latency: 1.2s"]:::metrics13. Quick Reference — The Whole Idea in One Diagram
One successful test ≠ Reliable prompt
Test dataset (varied, realistic inputs, including edge cases)
↓
Run current prompt against ALL of them
↓
Measure: accuracy, consistency, format compliance, cost, response time
↓
Specific failures -> specific diagnosis -> specific fix (Module 14)
↓
Re-run the FULL test set to confirm the fix helped (and didn't break
anything else)14. Prompts in Code — Calling an LLM
Here’s how prompt evaluation actually looks in code — running a test dataset against a prompt and computing real metrics.
Example 1 — Simple
A small test dataset run against a prompt, printing pass/fail for each case.
import anthropic
client = anthropic.Anthropic()
test_cases = [
{"input": "my order #4471 never showed up", "expected": "4471"},
{"input": "ORDER: 8823", "expected": "8823"},
]
for case in test_cases:
response = client.messages.create(
model="claude-sonnet-4-6", max_tokens=20,
messages=[{"role": "user", "content":
f"Extract the order number (digits only) from: {case['input']}"}]
)
actual = response.content[0].text.strip()
status = "PASS" if actual == case["expected"] else "FAIL"
print(f"[{status}] expected={case['expected']!r} actual={actual!r}")Example 2 — Intermediate
A slightly larger test dataset with an accuracy metric computed across all cases, including a genuine edge case.
import anthropic
client = anthropic.Anthropic()
test_cases = [
{"input": "my order #4471 never showed up", "expected": "4471"},
{"input": "ORDER: 8823", "expected": "8823"},
{"input": "no order number, just a shipping question", "expected": "NOT_FOUND"},
{"input": "order# 4471", "expected": "4471"}, # tricky formatting
]
def run_extraction(message: str) -> str:
response = client.messages.create(
model="claude-sonnet-4-6", max_tokens=20, temperature=0,
messages=[{"role": "user", "content":
f"Extract the order number (digits only) from this "
f"message. If none is mentioned, return NOT_FOUND: "
f"{message}"}]
)
return response.content[0].text.strip()
results = [{"input": c["input"], "expected": c["expected"],
"actual": run_extraction(c["input"])} for c in test_cases]
passed = sum(1 for r in results if r["actual"] == r["expected"])
print(f"Accuracy: {passed}/{len(results)} ({passed/len(results):.0%})")
for r in results:
if r["actual"] != r["expected"]:
print(f" FAILED: {r['input']!r} -> expected {r['expected']!r}, got {r['actual']!r}")Example 3 — Production Grade
A reusable evaluation harness that runs a test suite against any prompt function, reports accuracy, and saves results for tracking over time — the foundation of a real, ongoing evaluation pipeline.
import json
import anthropic
from datetime import datetime
client = anthropic.Anthropic()
def extract_order_number(message: str) -> str:
response = client.messages.create(
model="claude-sonnet-4-6", max_tokens=20, temperature=0,
messages=[{"role": "user", "content":
f"Extract the order number (digits only) from this "
f"message. If none is mentioned, return NOT_FOUND: "
f"{message}"}]
)
return response.content[0].text.strip()
def evaluate(prompt_function, test_cases: list, run_label: str) -> dict:
results = []
for case in test_cases:
actual = prompt_function(case["input"])
results.append({
"input": case["input"],
"expected": case["expected"],
"actual": actual,
"passed": actual == case["expected"],
})
accuracy = sum(r["passed"] for r in results) / len(results)
report = {
"run_label": run_label,
"timestamp": datetime.now().isoformat(),
"accuracy": round(accuracy, 3),
"total_cases": len(results),
"failures": [r for r in results if not r["passed"]],
}
return report
TEST_CASES = [
{"input": "my order #4471 never showed up", "expected": "4471"},
{"input": "ORDER: 8823", "expected": "8823"},
{"input": "no order number, just a shipping question", "expected": "NOT_FOUND"},
{"input": "order# 4471", "expected": "4471"},
]
report = evaluate(extract_order_number, TEST_CASES, run_label="v3-with-not-found-handling")
print(f"Accuracy: {report['accuracy']:.0%} ({report['total_cases']} cases)")
for f in report["failures"]:
print(f" FAILED: {f['input']!r} -> expected {f['expected']!r}, got {f['actual']!r}")
# Save the report for tracking accuracy across prompt versions over time
with open(f"eval_report_{report['run_label']}.json", "w") as f:
json.dump(report, f, indent=2)Saving each evaluation run’s report with a run_label is exactly what
makes this really useful over time — you can compare accuracy across
prompt versions (v1, v2, v3…) and directly see whether a specific
change actually improved reliability, precisely the practice Module 21
(Prompt Versioning) builds on next.
When to use it—and when not to
Use it when:
- prompts are reused or shipped to users.
- changes need evidence before deployment.
Do not rely on it when:
- one cherry-picked example is treated as an evaluation.
- a single metric hides important failure groups.
15. Interview Questions
Q: Why isn’t a single successful test enough to consider a prompt “good” or production-ready?
Ans: A single successful result only shows the prompt can work for that one specific input — it tells you nothing about how reliably it performs across the full range of varied, realistic inputs a real application will actually encounter, including edge cases, unusual formatting, and ambiguous scenarios. Reliability is a claim about performance across many inputs, not about any single result.
Q: What should a good test dataset for evaluating a prompt include, beyond typical, easy examples?
Ans: It should also include realistic edge cases (missing information, unusual formatting, unexpected input shapes) and really ambiguous cases where even a human might reasonably disagree on the ideal output. Testing only with easy, typical cases won’t reveal the failure modes that actually show up in real-world use, which are often exactly the cases that matter most for catching genuine reliability problems.
Q: How does evaluating an AI agent’s behavior differ from evaluating a single-response prompt?
Ans: Evaluating a single-response prompt typically focuses on whether the final output is correct and well-formatted. Evaluating an agent often needs to check the entire process leading to that output — for example, whether it followed required rules like searching before booking, or correctly reported a tool failure rather than fabricating a result — not just whether the final response looked reasonable in isolation.
Q: Why is it useful to save and compare evaluation results across different prompt versions over time, rather than just checking the current prompt’s accuracy once?
Ans: Tracking accuracy (and specific failure cases) across versions lets you directly verify whether a specific prompt change actually improved reliability, rather than just assuming it did based on how the change felt while writing it. It also helps catch regressions — cases where a change intended to fix one problem accidentally introduces a new failure elsewhere — which simply checking the current version in isolation wouldn’t reveal.
16. What You Should Remember
- A prompt that works once and a prompt that’s reliable are really different claims — only systematic evaluation across many inputs establishes the second one.
- A good test dataset includes typical cases, edge cases, and really ambiguous cases — not just the easy, expected examples.
- Evaluation should be ongoing and comparable across versions — saving results lets you verify a change actually helped, not just assume it did.
17. Quick Practice
For a prompt that classifies customer messages as “Urgent” or “Not Urgent,” write 5 test cases you’d want in your evaluation dataset, including at least one really ambiguous case.
18. Next Step
Next: Module 21 — Prompt Versioning and Templates — treating prompts like code: tracking changes, testing before deploying, and rolling back when something goes wrong.
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed