TechByteByByte

Section 14 — Testing

Test Spring AI prompts, model interactions and application behavior reliably.

Begin with the problem

Testing software whose wording can change

AI output is not always identical across runs, so exact sentence matching is often a weak test. A reliable test strategy separates deterministic Java logic from probabilistic model quality.

unit tests → contract tests → model integration tests → evaluation set

What you will learn

  • Test orchestration without a live model.
  • Test schemas and tool contracts deterministically.
  • Use evaluation datasets for behavior.
  • Control cost and flakiness in integration tests.

Current official reference: Spring AI documentation for this topic. The examples below primarily preserve the stated 1.1.x course target. Where Spring AI 2.0 differs, the text must treat that behavior as version-specific rather than universal.

(Continues from Section 13. Target: Spring AI 1.1.x / Spring Boot 3.5.x.)

14.1 Why AI Testing Needs a Different Mental Model

What’s really different about testing Spring AI code: the same input can legitimately produce different (but equally valid) outputs across calls, given real model non-determinism (recall temperature from the glossary — the model isn’t guaranteed to produce the exact same wording twice) — so a naive assertEquals(expectedString, actualResponse) is often the wrong test entirely. Spring AI testing splits into layers matching where determinism actually exists: deterministic unit tests around your orchestration code (mocked models), integration tests against real models with structural/semantic assertions rather than exact-match, and evaluation-based regression tests using the LLM-as-judge pattern from Section 7.

Real-world analogy — Testing a Restaurant, Not a Vending Machine: A vending machine test is exact-match: press B4, expect exactly one specific candy bar, every time. A restaurant test can’t be exact-match on the dish’s precise plating every time — instead you test process (did the kitchen follow food-safety procedure — this is your unit-tested orchestration logic) and quality against criteria (is the dish seasoned adequately, is it the right dish for what was ordered — this is your evaluation-based testing), not “is this literally byte-identical to yesterday’s plate.”

Analogy: The Flight Simulator vs. The Real Airplane Cockpit Imagine training a new commercial airline pilot for emergency landings:

  • The Real Cockpit (Production Remote API calls): You don’t put the trainee in a multi-million dollar passenger plane, fly into a thunderstorm, and cut the engine just to see if they turn the dial correctly (running remote frontier LLM APIs during local test runs). It’s incredibly slow, exposes you to unpredictable wind conditions (non-determinism), and leaves you with a massive fuel bill.
  • The Flight Simulator (WireMock / Mocks): You put them in a desktop simulator room. The controls feel exactly the same, but the weather parameters are completely stubbed (WireMock stubs, Mockito models). You run 10 landings in 10 minutes at zero real-world cost.
  • The Check Flight (LLM-as-a-judge): Periodically, an inspector flies in the cabin to evaluate the overall smoothness and safety of the flight pattern (evaluation tests checking prompt context recall).

📊 Visual Flowchart: Spring AI Test Suite Classification

Here is the split between Unit, Integration, and LLM-as-a-judge evaluation testing:

graph TD
    classDef unit fill:#2ecc71,stroke:#333,stroke-width:1px,color:#fff;
    classDef integration fill:#3498db,stroke:#333,stroke-width:1px,color:#fff;
    classDef eval fill:#9b59b6,stroke:#333,stroke-width:1px,color:#fff;

    TestRun["Start Test Suite"] --> Unit["1. Unit Tests<br>(Mock ChatModel via Mockito)"]:::unit
    Unit --> UnitVerify["Assert: advisor logic executed,<br>Tool methods invoke correctly"]:::unit

    TestRun --> Integration["2. Integration Tests<br>(WireMock stubbing connection endpoints)"]:::integration
    Integration --> IntegrationVerify["Assert: HTTP headers match,<br>JSON schemas render valid properties"]:::integration

    TestRun --> Eval["3. Evaluation Tests<br>(Call remote LLM-as-a-judge)"]:::eval
    Eval --> EvalVerify["Assert: answer correctness score > 0.85,<br>context grounding checked"]:::eval

14.2 Unit Testing — Mock the ChatModel, Test Your Orchestration

@ExtendWith(MockitoExtension.class)
class SupportServiceTest {

    @Mock
    private ChatModel chatModel;

    @Test
    void shouldEscalateWhenModelIndicatesUncertainty() {
        // Mock the ChatModel, not ChatClient directly — ChatClient is a
        // thin wrapper; mocking at the ChatModel level lets your Advisor
        // chain (memory, logging, custom advisors) run for real during
        // the test, which is usually what you actually want to verify
        ChatResponse mockResponse = new ChatResponse(List.of(
                new Generation(new AssistantMessage(
                        "I'm not confident I can answer this correctly."))));
        when(chatModel.call(any(Prompt.class))).thenReturn(mockResponse);

        ChatClient chatClient = ChatClient.builder(chatModel).build();
        SupportService service = new SupportService(chatClient);

        SupportResult result = service.handleQuery("complex edge case question");

        assertThat(result.wasEscalated()).isTrue();
    }
}

Key testing-architecture decision: mock ChatModel, not ChatClient. ChatClient is largely orchestration glue (Advisors, builders) that you generally want exercised in tests — mocking it directly bypasses memory advisors, logging advisors, custom business-logic advisors, which is usually exactly the logic you’re trying to verify actually works correctly, not the thing you want to stub away.

14.3 Testing Tool-Calling Logic in Isolation

Tool methods are ordinary Spring bean methods (Section 9) — test them directly without any model involved at all:

@Test
void refundToolShouldRejectAlreadyRefundedOrder() {
    RefundTools tools = new RefundTools(mockRefundService);
    when(mockRefundService.process("order-123"))
            .thenThrow(new RefundAlreadyProcessedException());

    RefundResult result = tools.processRefund("order-123");

    assertThat(result.isFailed()).isTrue();
    assertThat(result.message()).contains("already refunded");
}

This is standard unit testing with zero AI-specific concerns — worth stating explicitly because it’s easy to over-complicate tool testing by routing it through a full ChatClient call when a direct method test is faster, more reliable, and tests the actual business logic more precisely (whether the model chooses to call the tool correctly is a separate integration-level concern, covered next).

14.4 Integration Testing Against Real Models

@SpringBootTest
@EnabledIfEnvironmentVariable(named = "OPENAI_API_KEY", matches = ".+")
class RagIntegrationTest {

    @Autowired
    private ChatClient ragChatClient;

    @Test
    void shouldRetrieveAndCiteRelevantContext() {
        String response = ragChatClient.prompt()
                .user("What is our return policy for electronics?")
                .call()
                .content();

        // structural/semantic assertions, NOT exact-match —
        // this is the correct testing posture against a real,
        // non-deterministic model call
        assertThat(response).containsIgnoringCase("30 days"); // known
                                                                // fact from
                                                                // the test
                                                                // knowledge base
        assertThat(response).isNotBlank();
        assertThat(response.length()).isLessThan(2000); // sanity bound,
                                                          // catches runaway
                                                          // generation
    }
}

Real production pattern: gate real-model integration tests behind an environment variable (@EnabledIfEnvironmentVariable) so they run in CI where API keys are configured but don’t block local development when they’re absent, and separate them clearly (e.g., a @Tag("integration") + Surefire/Failsafe profile split) from fast unit tests so the full suite doesn’t require network access and real API cost on every local build.

14.5 Contract Testing for Tool Schemas and Structured Output

For tool-calling and structured-output contracts specifically — verifying the shape the model is asked to produce/consume stays stable across refactors, independent of whether a real model call succeeds:

@Test
void toolSchemaShouldRemainStable() {
    ToolCallback callback = ToolCallbacks.from(new WeatherTools(mockClient)).get(0);

    String schema = callback.getToolDefinition().inputSchema();

    // snapshot/contract test — fails loudly if a refactor accidentally
    // changes the generated schema shape the model depends on,
    // catching a class of bug that wouldn't surface until a real
    // model call started failing to produce valid arguments
    assertThat(schema).isEqualToIgnoringWhitespace(
            Files.readString(Path.of("src/test/resources/weather-tool-schema.json")));
}

This kind of snapshot/contract test catches an underappreciated failure mode: a seemingly-harmless refactor (renaming a parameter, changing a type from String to an enum) silently changes the JSON Schema presented to the model, which can degrade tool-calling reliability in production without any exception being thrown anywhere in your test suite — the code still compiles and runs, it just gets called with worse arguments by the model, or not called at all if the schema becomes confusing.

14.6 Load Testing

AI-backed endpoints have a fundamentally different load profile than typical CRUD endpoints: latency is dominated by the model provider (seconds, not milliseconds), and cost scales directly with request volume in a way a typical database-backed endpoint’s cost doesn’t. Standard tools apply (Gatling, k6, JMeter) with AI-specific considerations:

// k6 example — deliberately conservative concurrency given real API cost
import http from "k6/http";
import { check, sleep } from "k6";

export const options = {
  stages: [
    { duration: "2m", target: 10 }, // ramp to 10 concurrent users
    { duration: "5m", target: 10 }, // hold — enough to observe
    // rate-limiting/fallback
    // behavior under sustained load
    { duration: "1m", target: 0 },
  ],
};

export default function () {
  const res = http.post(
    "https://staging.example.com/chat",
    JSON.stringify({ message: "What is your return policy?" }),
    { headers: { "Content-Type": "application/json" } },
  );
  check(res, { "status is 200": (r) => r.status === 200 });
  sleep(1);
}

Load-test against a mocked/stubbed ChatModel for pure infrastructure capacity testing (can your service handle N concurrent connections, is your thread pool sized correctly) separately from testing against the real provider (which validates rate-limit handling, fallback engagement under provider degradation, and actual cost at target load) — conflating these two goals in one load test either wastes real API budget testing infrastructure concerns, or fails to validate real-provider behavior because a mock masked it.

14.7 Evaluation-Based Regression Testing

This directly extends Section 7’s RelevancyEvaluator into a CI-gate pattern — the closest thing to “exact-match testing” that’s actually appropriate for really non-deterministic model output:

@Test
void ragResponsesShouldRemainGroundedAcrossKnownQuestions() {
    List<TestCase> regressionSuite = loadRegressionSuite("rag-eval-suite.json");
    RelevancyEvaluator evaluator = new RelevancyEvaluator(chatClientBuilder);

    List<String> failures = new ArrayList<>();
    for (TestCase testCase : regressionSuite) {
        List<Document> retrieved = vectorStore.similaritySearch(testCase.question());
        String response = ragChatClient.prompt().user(testCase.question()).call().content();

        EvaluationResponse eval = evaluator.evaluate(
                new EvaluationRequest(testCase.question(), retrieved, response));

        if (!eval.isPass()) {
            failures.add(testCase.question() + ": " + eval.getFeedback());
        }
    }

    assertThat(failures).isEmpty();
}

Production nuance: because the evaluator itself is an LLM call, this test suite is (a) non-free to run and (b) itself has some non-determinism — run it as a scheduled/nightly CI gate rather than on every commit for cost/speed reasons, and treat isolated single-case failures with some skepticism (re-run before treating as a confirmed regression) while treating a trend of increasing failures across the suite as a strong, actionable signal (prompt template drift, upstream model version change, RAG ingestion quality regression).

14.8 Common Mistakes

  1. Exact-match assertions (assertEquals) against real model output — brittle, fails on entirely valid response variation; use structural/semantic/contains assertions instead.
  2. Mocking ChatClient directly instead of ChatModel, bypassing the Advisor chain logic that’s usually exactly what needs testing.
  3. No environment-gating on real-model integration tests, blocking local development without API keys or burning API cost on every commit.
  4. No schema/contract tests for tool definitions or structured output types, letting silent schema-shape regressions through refactors.
  5. Load testing against a real provider without separating infrastructure-capacity goals from real-provider-behavior goals, either wasting budget or missing real rate-limit/fallback validation.
  6. Running evaluation-based regression suites on every commit without considering their cost and non-determinism, versus a more appropriate scheduled cadence.

14.9 Debugging

For a flaky integration test against a real model, first determine whether the flakiness is (a) genuine model output variability (expected, fix the assertion to be less brittle) or (b) an actual intermittent failure (rate limiting, transient network issue, provider-side degradation) — log the full raw response and any exception on every failure, don’t just report pass/fail, since distinguishing these two causes changes whether you fix the test or investigate the system.

14.10 Interview Questions

  1. Why is exact-match assertion generally the wrong testing posture for real-model integration tests, and what should replace it?
  2. Why should unit tests mock ChatModel rather than ChatClient directly?
  3. How would you test tool-calling business logic without involving any actual model call?
  4. What’s the value of a schema/contract snapshot test for a tool definition, and what failure mode does it specifically catch?
  5. Describe the difference between load-testing infrastructure capacity versus load-testing real-provider behavior, and why they should be separated.
  6. Why is RelevancyEvaluator-based regression testing described as “the closest thing to exact-match testing” appropriate for non-deterministic model output?
  7. What’s the cost/non-determinism trade-off that argues for running evaluation-based regression suites on a scheduled cadence rather than every commit?
  8. How would you environment-gate real-model integration tests so they run in CI but don’t block local development without API keys?
  9. What’s the risk of a “harmless” refactor (parameter rename, type change) on tool-calling reliability, and how would a contract test catch it before production?
  10. Describe how you’d distinguish a really flaky integration test (model variability) from an actual intermittent system failure (rate limiting, network issue).
  11. Why does mocking at the ChatModel level preserve Advisor chain behavior in a unit test, and why does that matter for what you’re actually verifying?
  12. What sanity-bound assertions (beyond content-matching) are appropriate for a real-model integration test response?
  13. How would you structure a k6/Gatling load test to observe fallback-chain engagement under sustained load specifically?
  14. What’s the argument for treating a single evaluation-suite failure with skepticism versus treating a trend of failures as an actionable signal?
  15. Why is a typical CRUD endpoint’s load profile fundamentally different from an AI-backed endpoint’s, and how should that change your load-testing approach?
  16. How would you test that a custom Advisor’s chain.nextCall() is invoked exactly once, given the production risk of a forgotten call (Section 3)?
  17. What’s the correct way to structure a JUnit test suite so fast unit tests and slow/costly real-model integration tests are clearly separated?
  18. Why might testing tool method logic directly (bypassing the model entirely) be preferable to routing every tool test through a full ChatClient call?
  19. What observability data should be logged on every integration test failure to make root-causing feasible?
  20. How would you build a regression test dataset (question/expected-grounding pairs) for a RAG system, and what would trigger updating it over time?

14.11 Best Practices Checklist

  • Mock ChatModel, not ChatClient, in unit tests to preserve Advisor chain coverage.
  • Use structural/semantic assertions, never exact-match, against real-model integration test output.
  • Test tool business logic directly as plain Java method tests, independent of model involvement.
  • Add contract/snapshot tests for tool schemas and structured-output types.
  • Environment-gate real-model integration tests; separate them from the fast unit test suite.
  • Separate load-testing goals: infrastructure capacity (mocked model) versus real-provider behavior (real model, conservative concurrency).
  • Run evaluation-based regression suites on a scheduled cadence, treating trends as signal and isolated failures with appropriate skepticism.

14.12 Key Takeaways

  • Model non-determinism means testing strategy must be layered: deterministic unit tests around orchestration, structural assertions for real-model integration tests, and LLM-as-judge evaluation for genuine quality regression detection.
  • Mock at the ChatModel boundary, not ChatClient, to keep Advisor logic under test.
  • Contract/snapshot testing for tool schemas and structured-output shapes catches a real, otherwise-silent class of regression.
  • Load testing an AI-backed endpoint needs explicit separation between infrastructure-capacity and real-provider-behavior goals given cost and rate-limit realities.
  • Evaluation-based regression testing is powerful but non-free and itself non-deterministic — schedule and interpret it accordingly.

End of Section 14. Next: Section 15 — Security (Prompt Injection, Secrets, PII, Authorization, Authentication, Rate Limiting, Guardrails, Validation).

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed