AI Testing | TechByteByByte

Begin with the problem

AI testing needs two toolboxes: exact tests for deterministic code and evaluations for variable model behavior. A complete test plan tells you which toolbox belongs at each boundary.

unit/contract/integration tests + model/RAG/agent evals + load/security/chaos tests

What you will learn

Map every component to an appropriate test or evaluation method.
Test schemas, prompts, retrieval, agents, security, load, and failure handling.
Avoid exact-match assertions for open-ended generated text.

Current production grounding: OpenAI’s Evals documentation shows dataset- and grader-based evaluation for model applications.

These links document current public features and practices. They do not reveal a provider’s private implementation or guarantee that every product uses identical defaults.

1. The Engineering Problem

Module 2 established that traditional testing breaks down for probabilistic output, and Module 10 gave you evaluation as the replacement. This module closes the loop: what does a COMPLETE test suite for a real AI system actually look like, given that some parts are deterministic and testable in the traditional sense, and some need evaluation instead?

2. The Complete Testing Taxonomy

Test Type	What It Verifies	Deterministic?
Unit tests	Input validation, parsing, retry/timeout control flow (Module 9, 14)	✅ Yes
Integration tests	Components work together (retrieval → orchestration → model call)	✅ Yes (the wiring; not the model’s output)
Contract tests	A structured-output schema is well-formed and stable (Module 9)	✅ Yes
Prompt tests	A rendered prompt template compiles with all required variables (Module 5)	✅ Yes
Model/Evaluation tests	Whether the model’s actual response is faithful, relevant, correct (Module 10)	❌ No — needs evaluation
RAG tests	Retrieval precision/recall against a golden set (your RAG course)	❌ No — needs evaluation
Agent tests	Task completion rate, tool-call accuracy (your Agents course)	❌ No — needs evaluation
Regression tests	Evaluation SCORES haven’t dropped vs. baseline (Module 11)	❌ No — score-based, not exact-match
Load tests	System behavior under concurrent volume (latency, error rate)	✅ Yes
Security tests	Injection defenses, tenant isolation hold (Module 13)	✅ Yes (the defense mechanism itself)
Chaos tests	The system degrades gracefully when a dependency fails (Module 14)	✅ Yes (the failure-handling logic)

3. The Determinism Boundary — Restated Precisely

DETERMINISTIC (traditional testing applies):

  - Code that runs BEFORE the model is called (validation, retrieval
    filtering, prompt construction)
  - Code that runs AFTER the model responds (parsing, schema
    validation, error handling)
  - Infrastructure behavior (retries, timeouts, circuit breakers,
    load handling)

NOT deterministic (evaluation applies, Module 10):

  - The model's actual GENERATED content
  - Whether a RAG system retrieved the RIGHT documents for
    a given query
  - Whether an agent completed a task correctly

This is the exact same boundary Module 2 introduced — this module’s contribution is mapping it onto a COMPLETE, practical test suite structure, showing precisely where each test type belongs.

4. Contract Testing — An Important, Often-Missed Layer

A CONTRACT TEST verifies the SHAPE of structured output stays
STABLE -- e.g., a `ticket_category` field remains a
string, `urgency_score` remains an integer -- independent
of whether the SPECIFIC values the model generates are correct.

This is directly Module 9's validation logic, TESTED as its own
deterministic contract, separate from whether the model's
JUDGMENT (the actual category chosen) is correct.

5. Chaos Testing — Module 14, Made Testable

Module 14's reliability patterns: a CHAOS TEST deliberately
INJECTS a failure (kill the model provider connection mid-test,
simulate a vector DB timeout) and asserts the system degrades gracefully -- falls back correctly, doesn't hang, doesn't
crash -- rather than simply HOPING the reliability code works when a
real outage eventually happens.

6. A Real-World Analogy — The Security Checkpoint, Once More

Module 13's checkpoint analogy: testing a SECURITY
CHECKPOINT doesn't mean checking whether EVERY possible person who
could walk through is a security threat (that's
unknowable in advance) -- it means testing whether the
CHECKPOINT ITSELF correctly identifies and blocks a KNOWN test case
of a threat pattern.

Similarly, testing an AI system's SECURITY defenses (Section 2)
means testing whether the DEFENSE MECHANISM correctly blocks known
attack patterns -- not whether the MODEL will never be manipulated
by some novel, unknown attack.

7. A worked developer example

TechCorp’s complete test suite for their support assistant, mapped to Section 2’s taxonomy:

Layer	Test	Type
Input validation	Rejects malformed requests	Unit test
Structured output	`urgency_score` is always an integer	Contract test
Retrieval	Precision@5 stays above 0.85 on golden set	RAG evaluation test
Response quality	Faithfulness score stays above baseline	Regression test
Reliability	Circuit breaker opens correctly when provider fails	Chaos test
Security	Injection pattern in test input is blocked	Security test
Scale	System handles 500 concurrent requests within latency budget	Load test

8. How Is This Used in the Industry?

🤖 How Is This Used in the Industry?

Production AI teams maintain two distinct test suites running on different cadences — a fast, traditional CI suite (unit/contract/integration tests) gating every code change, and a slower evaluation suite (Module 10-11) gating every prompt/model/ retrieval change, exactly reflecting Section 3’s determinism boundary in real, practical tooling.

9. Common Mistakes

Incorrect idea: Writing exact-match assertions for model-generated content.

Why it is incorrect: This shortcut removes a validation or control boundary, allowing an error to pass into later stages where it becomes harder and more expensive to detect. As shown directly in Module 2, this fails even for correct responses.

Incorrect idea: Skipping contract tests because “the model’s output validation already covers it.”

Why it is incorrect: As shown directly in Section 4, contract tests catch schema drift independently of judgment correctness.

Incorrect idea: Never running chaos tests, only hoping reliability code (Module 14) works during a real outage.

Why it is incorrect: As shown directly in Section 5, testing failure-handling logic before a real incident is far cheaper than discovering a gap during one.

10. Code — A Test Classification Catalog

What this shows: a working catalog classifying test types by whether they’re deterministic — exactly Section 2’s table made into queryable, reference data a real team could use when deciding where a new test belongs.

from dataclasses import dataclass
from enum import Enum

class TestType(Enum):
    UNIT = "unit_test"
    CONTRACT = "contract_test"
    PROMPT = "prompt_test"
    EVALUATION = "evaluation_test"
    LOAD = "load_test"

class Determinism(Enum):
    FULLY_DETERMINISTIC = "fully_deterministic"
    NOT_DETERMINISTIC = "not_deterministic"

@dataclass
class TestClassification:
    test_type: TestType
    determinism: Determinism
    what_it_verifies: str

# Section 2's table, made into structured, queryable data
TEST_CATALOG = [
    TestClassification(TestType.UNIT, Determinism.FULLY_DETERMINISTIC,
                        "Input validation, parsing logic, retry/timeout control flow"),
    TestClassification(TestType.CONTRACT, Determinism.FULLY_DETERMINISTIC,
                        "The structured-output schema itself is well-formed and matches expectations"),
    TestClassification(TestType.PROMPT, Determinism.FULLY_DETERMINISTIC,
                        "The rendered prompt template compiles correctly with all required variables"),
    TestClassification(TestType.EVALUATION, Determinism.NOT_DETERMINISTIC,
                        "Whether the MODEL'S generated response is faithful, relevant, correct"),
    TestClassification(TestType.LOAD, Determinism.FULLY_DETERMINISTIC,
                        "System behavior (latency, error rate) under concurrent request volume"),
]

def what_can_be_deterministic(test_type: TestType) -> TestClassification:
    """Directly implements Section 3's determinism boundary -- a
    quick, reference for where a new test type belongs."""
    for t in TEST_CATALOG:
        if t.test_type == test_type:
            return t
    return None

for t in [TestType.UNIT, TestType.EVALUATION, TestType.CONTRACT]:
    result = what_can_be_deterministic(t)
    print(f"[{result.test_type.value}] deterministic={result.determinism.value}")
    print(f"  Verifies: {result.what_it_verifies}\n")

Expected Output:

[unit_test] deterministic=fully_deterministic
  Verifies: Input validation, parsing logic, retry/timeout control
flow

[evaluation_test] deterministic=not_deterministic
  Verifies: Whether the MODEL'S generated response is faithful, relevant, correct

[contract_test] deterministic=fully_deterministic
  Verifies: The structured-output schema itself is well-formed and
matches expectations

What this confirms: unit and contract tests are correctly classified as fully deterministic, while evaluation tests are correctly classified as not — exactly Section 3’s boundary, made into a queryable reference a real team could consult when adding a new test to their suite, rather than re-deriving the classification each time.

11. Production Considerations

Run deterministic test suites (unit, contract, integration) on every commit — they’re fast and cheap
Run evaluation suites (Module 10-11) on every prompt/model/retrieval change specifically — slower, but necessary before those changes deploy

12. Trade-offs

Chaos testing requires deliberate failure injection infrastructure — real setup cost, worthwhile for the confidence it provides before a real outage occurs
Contract tests add maintenance overhead as schemas evolve — worthwhile for catching schema drift that would otherwise silently break downstream consumers

13. Chapter Summary

A complete AI system test suite spans both traditional, deterministic tests (unit, contract, integration, load, security, chaos — everything around the model call) and evaluation-based tests (model/RAG/agent quality, regression — everything about the model’s actual output).

The skill this module teaches is correctly classifying which category a given test belongs to and building two distinct, appropriately-paced test suites rather than either forcing exact-match assertions onto probabilistic output or abandoning traditional testing discipline for the deterministic parts of the system.

14. Visual Cheat Sheet

DETERMINISTIC (traditional CI tests, every commit):
  unit + contract + integration + load + security + chaos

NOT DETERMINISTIC (evaluation suite, every prompt/model/retrieval
change):
  model quality + RAG quality + agent quality + regression

15. Top Takeaways

A complete AI test suite spans both deterministic tests (around the model) and evaluation-based tests (about the model’s output).
Contract tests verify structured-output SHAPE stability, independent of whether the model’s specific judgment is correct.
Chaos tests deliberately inject failures to verify Module 14’s reliability patterns work, before a real outage.
Security tests verify defense mechanisms correctly block known attack patterns — not that the model can never be manipulated by any unknown attack.
Deterministic and evaluation suites run on different cadences — fast CI tests on every commit, slower evaluation on every quality-affecting change.

16. Interview Questions

Q: 1. Design a complete test suite structure for a production RAG system, distinguishing what runs on every commit versus what runs before a prompt or retrieval change deploys.**

Ans: On every commit: unit tests (validation, parsing), contract tests (structured-output schema stability), integration tests (components wire together correctly), and security tests (injection defenses hold) — all deterministic and fast.

Before any prompt, model, or retrieval configuration change deploys: the evaluation suite (Module 10-11) — retrieval precision/recall against a golden dataset, faithfulness/relevance scoring, and regression comparison against the current baseline — slower but necessary since these changes affect probabilistic output quality.

Why it matters: This structure directly reflects the determinism boundary (Module 2) in practical, deployable tooling.
Real-world example: Section 7’s TechCorp test suite mapping.
Common mistake: Running the full evaluation suite on every commit regardless of whether the change could affect model output, wasting CI time.
Interviewer is testing: Whether the candidate can structure a practical, two-speed test suite rather than treating all tests uniformly.
Likely follow-up: “What would trigger the evaluation suite specifically?” → Any change touching prompts, model selection, retrieval configuration, or context assembly — anything that could affect the model’s actual output.

Q: 2. What is a contract test in the context of an AI system, and why is it useful independent of evaluation?**

Ans: A contract test verifies that structured output’s SHAPE — field names, types — stays stable, independent of whether the specific values the model generates are judged correct.

It’s useful because schema drift (a field silently becoming a string instead of an integer, for example) is a deterministic bug that evaluation (which focuses on content quality) might not catch, and that would break downstream code relying on that schema.

Why it matters: Without contract tests, a schema regression could slip through if evaluation only checks whether the VALUES seem reasonable, not whether the STRUCTURE is correct.
Real-world example: Section 4’s urgency_score example.
Common mistake: Assuming evaluation tests alone are sufficient coverage for structured-output reliability.
Interviewer is testing: Whether the candidate recognizes a distinct, deterministic testing layer beyond evaluation.
Likely follow-up: “How would you implement a contract test for a specific schema?” → Module 9’s validation function, tested with fixed, known-good and known-bad inputs, exactly like any traditional unit test.

17. Scenario-Based Question

Scenario: TechCorp’s team ships a prompt change that passes their evaluation suite (faithfulness and relevance scores both look good).

Two days later, a downstream service that parses the assistant’s structured output starts throwing errors — the model’s new phrasing style caused it to occasionally omit the requires_escalation field entirely, something the evaluation suite’s content-quality scoring never checked.

Problem Analysis: Section 4 and 9’s point — evaluation covers content QUALITY, not structural CONTRACT stability; these are different concerns requiring different tests.
How to Think: The prompt change was evaluated for the wrong thing in isolation — content quality passed, but a structural regression slipped through because no contract test existed to catch it.
Investigation: Confirm the missing field is a new occurrence correlated with the prompt change, and that no contract test currently exists for this specific schema.
Root Cause: No contract test verifying requires_escalation is always present and boolean — Section 9’s common mistake, realized in practice.
Solution: Add a contract test (Section 4, Module 9’s validation logic) to the deterministic CI suite, catching this class of regression on every future commit — not just relying on the evaluation suite’s content-focused scoring.
Trade-offs: Adding contract tests for every structured-output field requires, upfront test-writing effort — a real, worthwhile investment given the alternative is exactly this kind of downstream production break.
Production Considerations: This scenario directly demonstrates Section 13’s core point — a complete test suite needs BOTH evaluation (content quality) AND contract tests (structural stability), since passing one provides no guarantee about the other.

18. Next Step

Next: Module 25 — Deployment — local, Docker, cloud, serverless, and GPU deployment options, hosted vs. self-hosted serving, and blue-green/canary deployment strategies.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026