TechByteByByte

Section 11 — Structured Output

Generate typed, structured responses from AI models using Spring AI converters.

Begin with the problem

Turning model text into safe Java data

A paragraph is easy for a person to read but awkward for code to process. Structured output asks for a known shape and then converts and validates it before the application trusts it.

model response → schema check → converter → Java object

What you will learn

  • Convert responses into Java types.
  • Distinguish prompting for JSON from provider-enforced schemas.
  • Handle parsing and validation failures.
  • Know when free-form text is the better output.

Current official reference: Spring AI documentation for this topic. The examples below primarily preserve the stated 1.1.x course target. Where Spring AI 2.0 differs, the text must treat that behavior as version-specific rather than universal.

(Continues from Section 10. Target: Spring AI 1.1.x / Spring Boot 3.5.x.)

11.1 Why Structured Output Is Two Problems, Not One

Current 2.0 behavior to know first

Spring AI 2.0 can use provider-native structured output when the selected provider and model support it. That is stronger than merely adding “return JSON” instructions to a prompt, but it still does not remove the need to validate business rules after parsing.

Java type or JSON schema
          ↓
provider constraint (when supported)
          ↓
model output → parse → business validation → trusted application data

If provider-native enforcement is unavailable, a StructuredOutputConverter can still place format instructions in the prompt and convert the returned text. Treat these as two different levels of guarantee, not as interchangeable names for the same feature.

Beginner primer: LLMs fundamentally generate free-form text, not typed data. “Structured output” means getting the model to produce text that’s actually valid JSON matching a specific shape (say, matching your OrderExtraction record), so your application can parse it directly into a real Java object instead of trying to regex-scrape meaning out of a prose paragraph.

“Get JSON back from the model” is actually two distinct engineering problems Spring AI solves separately: (1) instructing the model to produce output matching a schema (prompt-level, sometimes reinforced with provider-level constrained decoding), and (2) reliably converting the model’s raw text response into a typed Java object, including handling the cases where the model’s output doesn’t quite match. StructuredOutputConverter addresses both.

Real-world analogy — Government Form Processing: Problem 1 is designing a form with clear labeled fields (the schema/instructions) so people fill it out correctly. Problem 2 is the data-entry clerk who still has to handle the citizen who wrote “N/A” in a numeric field, or added extra commentary in the margins — parsing real-world (model) output that’s usually well-formed but needs defensive handling, not blind JSON.parse().

Analogy: The Government Form Processing (Form & Clerk) Think of getting structured JSON back from an LLM as a two-stage form processing workflow:

  • The Form Designer (Problem 1 - The Schema): You design a clean tax return layout with strict fields: “Name: [text], Age: [number], Salary: [number]” (JSON Schema instructions). You print this form template (Prompt Injection) and hand it to the citizen (the LLM).
  • The Data Clerk (Problem 2 - The Parser): Even if the form has strict boxes, citizens write stray comments in the margins, like “See attachment” or put dollar signs inside numeric fields.
  • The Data-Entry Clerk (Spring AI BeanOutputConverter) doesn’t blindly import the paper. They strip out formatting fences (json ... ), ignore margins commentary, cast values, and handle validation errors dynamically.

📊 Visual Flowchart: Structured Output Conversion & Parsing Pipeline

Here is how target records are translated to schemas, sent as instructions, parsed, and validated:

graph TD
    classDef fail stroke:#e74c3c,stroke-width:2px;

    Record["Java Record: OrderExtraction.class"] --> Schema["1. BeanOutputConverter<br>(Generate schema + prompt instruction string)"]
    Schema --> Prompt["2. Inject instruction into Prompt userTurn text"]
    Prompt --> Call["3. ChatClient: execute call()"]

    Call --> RawText["4. Receive Raw Response Text<br>(e.g. ```json {'orderId': '123'} ```)"]

    RawText --> Filter["5. Clean Response<br>(Strip markdown JSON fences)"]
    Filter --> Jackson{"6. Jackson ObjectMapper:<br>parse clean JSON"}

    Jackson -->|Parse Success| Obj["7. Java Record Instance"]
    Jackson -->|Parse Fail| Error["Throw StructuredOutputException"]:::fail

    Obj --> BusinessValidation{"8. Validator: run constraints<br>(e.g. @Positive, @Past)"}

    BusinessValidation -->|Pass| Output["Return Typed Record T"]
    BusinessValidation -->|Fail| ValidationError["Throw ExtractionValidationException<br>(Retry / feedback loop)"]:::fail

11.2 BeanOutputConverter — The Core Mechanism

public record OrderExtraction(
        String orderId,
        BigDecimal amount,
        LocalDate orderDate,
        OrderStatus status,
        List<String> itemSkus
) {}

public enum OrderStatus { PENDING, SHIPPED, DELIVERED, CANCELLED }
OrderExtraction result = chatClient.prompt()
        .user("Extract order details from this email: " + emailText)
        .call()
        .entity(OrderExtraction.class);

11.2.1 Internal Mechanism

.entity(OrderExtraction.class)
        │
        ▼
1. BeanOutputConverter<OrderExtraction> constructed, which:
   - Generates a JSON Schema from the record's fields (same underlying
     schema-generation approach used by tool calling, Section 9 —
     Spring AI reuses one schema-generation mechanism across both
     structured output and tool parameter typing)
   - Produces a format instruction string appended to the prompt:
     "Your response should be in JSON format that adheres to the
      following schema: {...}. Do not include markdown code fences."
        │
        ▼
2. This instruction is injected into the Prompt (typically appended
   to the user message, or system message depending on API usage)
   BEFORE the call is made — this is why .entity() actually MODIFIES
   the request, not just the response parsing
        │
        ▼
3. Model responds with (hopefully) valid JSON text
        │
        ▼
4. BeanOutputConverter.convert(rawResponseText):
   - Strips markdown code fences if present (models frequently wrap
     JSON in ```json ... ``` despite instructions not to — this
     defensive stripping is built in, not something you write yourself)
   - Jackson ObjectMapper.readValue() deserializes into OrderExtraction
        │
        ▼
5. Typed object returned, or a StructuredOutputException thrown if
   parsing really fails (malformed JSON, schema violation Jackson
   can't coerce past)

Critical operational fact: .entity(Class<T>) changes what gets sent to the model, not just how the response is parsed. This matters for prompt-length/token budgeting — the schema description itself consumes context tokens, and for very large/deeply nested types, that overhead is non-trivial and worth measuring, not assumed negligible.

11.3 Records, Enums, and Nested Types

Records are the idiomatic choice — immutable, concise, and Spring AI’s schema generator handles them cleanly including nested records:

public record Invoice(
        String invoiceNumber,
        Vendor vendor,
        List<LineItem> lineItems,
        BigDecimal totalAmount
) {}

public record Vendor(String name, String taxId) {}
public record LineItem(String description, int quantity, BigDecimal unitPrice) {}

Enums map cleanly to JSON Schema enum constraints, which meaningfully improves reliability versus a free-text String status field — the model is given an explicit closed set of valid values rather than inferring acceptable strings, reducing the “SHIPED” (typo) or “Shipped” (wrong casing) class of near-miss errors that a plain string field invites.

List/Collection handling: List<T> works well for homogeneous collections; avoid Map<String, T> for structured extraction where the keys are meant to be meaningful (e.g., field names) — models handle “produce a list of objects, each with a name field” far more reliably than “produce a map keyed by name,” because the latter is a less natural JSON-generation pattern for most models to reason about consistently.

11.4 ListOutputConverter and MapOutputConverter

For simpler shapes than a full custom type:

List<String> topics = chatClient.prompt()
        .user("List the 5 main topics discussed in this transcript, one per line.")
        .call()
        .entity(new ParameterizedTypeReference<List<String>>() {});

Map<String, Object> genericExtraction = chatClient.prompt()
        .user("Extract key-value facts from this document")
        .call()
        .entity(new ParameterizedTypeReference<Map<String, Object>>() {});

Map<String, Object> extraction is useful for really dynamic/unpredictable schemas (exploratory extraction where you don’t know field names ahead of time) but sacrifices the type-safety and validation benefits of a concrete record type — use it deliberately, not as a default because it’s “easier,” since it pushes all shape validation to runtime code you now have to write yourself.

11.5 Schema Validation Beyond Jackson Deserialization

BeanOutputConverter gets you type-correct deserialization, but not business-rule validation. Layer standard Bean Validation on top:

public record OrderExtraction(
        @NotBlank String orderId,
        @Positive BigDecimal amount,
        @PastOrPresent LocalDate orderDate,
        @NotNull OrderStatus status
) {}
OrderExtraction result = chatClient.prompt()
        .user(extractionPrompt)
        .call()
        .entity(OrderExtraction.class);

Set<ConstraintViolation<OrderExtraction>> violations = validator.validate(result);
if (!violations.isEmpty()) {
    // model produced structurally valid JSON matching the schema,
    // but semantically invalid data (negative amount, future order
    // date) — this is a DIFFERENT failure class than a parse error,
    // and needs different handling: likely a retry with the violation
    // fed back to the model, not a generic error
    throw new ExtractionValidationException(violations);
}

This two-layer validation (schema-level via Jackson/JSON Schema, business-rule-level via Bean Validation) is the production-correct pattern — conflating them (assuming “it parsed” means “it’s correct”) is a common source of silently-wrong extracted data reaching downstream systems.

11.6 Error Recovery — What to Do When Parsing Fails

public <T> T extractWithRetry(ChatClient chatClient, String userMessage, Class<T> targetType,
                               int maxAttempts) {
    Exception lastException = null;

    for (int attempt = 1; attempt <= maxAttempts; attempt++) {
        try {
            return chatClient.prompt()
                    .user(u -> {
                        u.text(userMessage);
                        if (attempt > 1) {
                            // feed the PRIOR failure back to the model —
                            // this measurably improves success rate on retry
                            // versus a blind identical re-attempt
                            u.text(userMessage + "\n\nYour previous response could "
                                    + "not be parsed as valid JSON matching the "
                                    + "required schema. Please respond with ONLY "
                                    + "valid JSON, no markdown formatting, no "
                                    + "additional commentary.");
                        }
                    })
                    .call()
                    .entity(targetType);
        } catch (StructuredOutputException e) {
            lastException = e;
            log.warn("Structured output parse attempt {} failed: {}", attempt, e.getMessage());
        }
    }
    throw new ExtractionFailedException(
            "Failed to extract structured output after " + maxAttempts + " attempts", lastException);
}

Production-grade error recovery has three tiers, roughly in order of preference:

  1. Retry with the failure fed back to the model (shown above) — usually resolves transient formatting slip-ups (stray commentary, markdown fences the model added despite instructions).
  2. Fallback to a more lenient parse — e.g., regex-extracting a JSON substring from a response that includes extra prose around valid JSON, before giving up entirely.
  3. Fail loudly with full context logged (raw response text, schema used, attempt count) rather than silently returning a default/null object — a silently-defaulted extraction is far more dangerous downstream than a clear failure, especially for anything feeding a business process (one of this series’ worked applications — the customer support agent — is a real example of exactly this risk).

11.7 Provider-Level JSON Mode / Constrained Decoding

Some providers support a stronger guarantee than prompt-instruction-based JSON formatting — actual constrained decoding at the token-sampling level (OpenAI’s response_format: json_schema with strict mode, for instance), which meaningfully reduces (though for full JSON Schema strict mode, can effectively eliminate for supported providers/models) the parse-failure rate versus prompt-instruction alone:

OpenAiChatOptions options = OpenAiChatOptions.builder()
        .model("gpt-4o")
        .responseFormat(ResponseFormat.builder()
                .type(ResponseFormat.Type.JSON_SCHEMA)
                .jsonSchema(schemaFromClass(OrderExtraction.class))
                .build())
        .build();

This is provider-specific (Section 4’s leaky-portability point applies directly) — not every provider/model supports true constrained JSON decoding, and where it’s not supported, you’re back to prompt-instruction-based formatting with the full error-recovery tier from §11.6 as your defense. Check current provider capability before assuming strict-mode JSON schema enforcement is universally available; this is one of the faster-moving capability surfaces across providers.

11.8 Common Mistakes

  1. Assuming .entity() parsing never fails — no retry/error-recovery tier at all, letting a StructuredOutputException propagate as an unhandled 500.
  2. Conflating schema validity with business-rule validity — not layering Bean Validation on top of successful deserialization.
  3. Using Map<String, Object> by default instead of a concrete typed record, sacrificing compile-time safety for no real benefit when the schema is actually known ahead of time.
  4. Ignoring token-budget cost of large/deeply nested schema descriptions injected into every request.
  5. Not leveraging provider-level constrained decoding (JSON mode/strict schema) where available, relying entirely on prompt-instruction formatting when a stronger guarantee exists.
  6. Silently defaulting on extraction failure instead of failing loudly with full diagnostic context.

11.9 Debugging

logging:
  level:
    org.springframework.ai.converter: DEBUG

Always log the raw model response text alongside any StructuredOutputException — the exception message alone rarely tells you why parsing failed as clearly as seeing the actual malformed output (stray commentary, wrong field name the model invented, truncated JSON from hitting a token limit mid-generation — this last one specifically diagnosable by checking ChatResponseMetadata.getUsage() for whether finishReason was length rather than stop).

11.10 Interview Questions

  1. What are the two distinct problems StructuredOutputConverter solves, and why does separating them matter?
  2. How does .entity(Class<T>) modify the outgoing request, not just the response parsing?
  3. Why do records with enums generally produce more reliable structured extraction than free-text string fields for closed-set values?
  4. What’s the risk of using Map<String, Object> as a default extraction target instead of a concrete typed record?
  5. Describe the three-tier error recovery strategy for structured output parsing failures, in order of preference.
  6. Why does feeding the prior parse failure back to the model on retry measurably improve success rate over a blind re-attempt?
  7. What’s the difference between schema-level validity and business-rule validity, and why does successful Jackson deserialization not guarantee the latter?
  8. How would you diagnose a StructuredOutputException caused by the model’s response being truncated mid-generation due to a token limit?
  9. What provider-level capability offers a stronger guarantee than prompt-instruction- based JSON formatting, and why isn’t it universally available across providers?
  10. Why should Map<String, T> generally be avoided for structured extraction where field names are meant to be meaningful, versus List<T>?
  11. What defensive parsing behavior does BeanOutputConverter perform automatically regarding markdown code fences?
  12. How does Spring AI’s schema-generation mechanism for structured output relate to the mechanism used for tool-calling parameter schemas (Section 9)?
  13. What’s the token-budget cost consideration for very large or deeply nested output schema types, and how would you measure it?
  14. Why is silently defaulting to a null/empty object on extraction failure more dangerous than failing loudly, particularly for a business-process-feeding extraction?
  15. How would you layer Bean Validation on top of a successfully-parsed structured output record?
  16. What’s the practical difference between entity(Class<T>) and entity(ParameterizedTypeReference<T>)?
  17. Describe how you’d implement a fallback regex-based JSON extraction as a second-tier recovery strategy for a response containing valid JSON embedded in extra prose.
  18. Why might strict-mode constrained decoding not be available for every model even within the same provider’s lineup?
  19. What information should be logged alongside a structured-output parse failure to make it actually debuggable?
  20. How would you design an extraction pipeline where an extracted OrderExtraction’s business-rule violations (e.g., negative amount) trigger a targeted retry rather than a generic failure?

11.11 Best Practices Checklist

  • Always implement multi-tier error recovery (retry-with-feedback, lenient fallback parse, loud failure) rather than assuming .entity() always succeeds.
  • Layer Bean Validation on top of successful deserialization for business-rule checks Jackson/JSON Schema can’t express.
  • Prefer concrete typed records over Map<String, Object> whenever the schema is known ahead of time.
  • Use enums for closed-set fields instead of free-text strings.
  • Check and use provider-level constrained JSON decoding where the target model/provider supports it.
  • Log raw model response text alongside every structured-output parse failure.
  • Measure token overhead of large output schemas rather than assuming it’s negligible.

11.12 Key Takeaways

  • Structured output is prompt-injection-of-schema plus defensive response parsing — two separate concerns solved together by BeanOutputConverter.
  • Successful deserialization is not the same as business-rule-valid data; layer both validation levels deliberately.
  • Retry-with-feedback (showing the model its own parse failure) is meaningfully more effective than blind retry.
  • Provider-level constrained decoding, where available, is a stronger guarantee than prompt-instruction formatting — use it when the provider supports it.
  • Never silently default on extraction failure; fail loudly with full diagnostic context, especially when output feeds a downstream business process.

End of Section 11. Next: Section 12 — Streaming (SSE, Reactive, WebFlux, Token Streaming, Cancellation, Backpressure, Monitoring).

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed