Section 17 — Enterprise Architecture

Begin with the problem

Putting the pieces into a production system

A demo has one user and one model call. An enterprise system adds tenants, permissions, budgets, fallbacks, audit trails, shared data, scaling, and failure recovery.

gateway → policy → Spring AI pipeline → providers/tools/data → telemetry

What you will learn

• Place Spring AI in a larger system.
• Compare embedded and shared-gateway designs.
• Plan tenant isolation, resilience, and governance.
• Connect security, testing, cost, and observability.

Current official reference: Spring AI documentation for this topic. The examples below primarily preserve the stated 1.1.x course target. Where Spring AI 2.0 differs, the text must treat that behavior as version-specific rather than universal.

(Continues from Section 16. Target: Spring AI 1.1.x / Spring Boot 3.5.x. This section closes out the 17-section core curriculum before the worked applications.)

Analogy: The Automated Car Assembly Line Think of designing an enterprise-grade AI system architecture as planning a high-speed, modern automobile factory assembly line:

• The Mono-Mechanic (Embedded Per-Service): A single mechanic builds a car from raw steel in a single bay (each microservice embeds Spring AI, manages its own API keys, configures its own logging, and maintains its own vector database catalog). It is fast to start but impossible to standardize cost, safety, or quality across 20 mechanics.
• The Assembly Line (Dedicated AI Gateway / Shared Library): A standardized, moving conveyor belt:
  • Station 1 (API Gateway): Validates access badges and regulates traffic flow (Rate limiting & Circuit breakers).
  • Station 2 (Advisor Pipeline): Pulls passenger memory records, checks safety belts, and filters hazards (Memory retrieval & input guardrails).
  • Station 3 (Assembly Robot): Swaps tires or fits the battery (ChatModel calling different remote model endpoints).
  • Station 4 (Quality Control Inspector): Measures alignment specifications (Structured output parsing & schema validation).

📊 Visual Chart: End-to-End Enterprise Multi-Tenant AI Pipeline

Here is the complete path of a tenant-scoped request routing through distributed systems, RAG stores, and model APIs:

graph TD
    classDef secure fill:#2ecc71,stroke:#333,stroke-width:1px,color:#fff;
    classDef storage fill:#3498db,stroke:#333,stroke-width:1px,color:#fff;
    classDef gate fill:#f1c40f,stroke:#333,stroke-width:1px,color:#fff;

    UserReq["User Prompt + session JWT"] --> Gateway["1. Enterprise API Gateway<br>(Rate limits & Tenant verification)"]:::gate

    Gateway --> TenantRouter{"2. Tenant Routing filter"}

    subgraph Microservices [Spring Boot App Domain]
        TenantRouter -->|Tenant A| AppInstanceA["3. Service instance A<br>(Injects Tenant A API keys)"]:::secure
        TenantRouter -->|Tenant B| AppInstanceB["3. Service instance B"]
    end

    AppInstanceA --> RAGQuery["4. RAG Ingestion & Vector Retrieval"]

    subgraph Storage [Secure Persistence Tier]
        RAGQuery --> PGVector["5. PGVector DB Table<br>(WHERE tenantId = 'TenantA')"]:::storage
        AppInstanceA --> ChatMemory["6. Redis Session Cache<br>(ChatMemoryRepository TTL storage)"]:::storage
    end

    PGVector --> ContextEnriched["7. Augmented Context Prompt"]
    ChatMemory --> ContextEnriched

    ContextEnriched --> Observability["8. Micrometer Observation Convention<br>(Trace Span propagation)"]:::gate

    Observability --> ModelRouter{"9. Task Routing Selector"}

    ModelRouter -->|Reasoning| PrimaryModel["10. Primary Provider API"]:::secure
    ModelRouter -->|Fallback| SecondaryModel["10. Fallback Provider API"]

---

17.1 Where Spring AI Sits in a Microservices Topology

The specific question this section answers: where does the AI capability live — as a dedicated “AI service” other services call, embedded directly inside each domain service that needs it, or some hybrid — and what does that choice cost you architecturally.

Pattern A: Dedicated AI Gateway Service

┌──────────┐     ┌──────────┐     ┌──────────────────┐    ┌──────────┐
│  Order    │────▶│   API     │────▶│   AI Service      │────▶│ Provider │
│  Service  │     │  Gateway  │     │ (Spring AI here)  │     │ (OpenAI) │
└──────────┘     └──────────┘     └──────────────────┘    └──────────┘
     ▲                                       │
     └─────────── async event/callback ──────┘

Pattern B: Embedded Per-Domain-Service

┌────────────────────────┐     ┌──────────┐
│  Order Service           │────▶│ Provider │
│  (Spring AI embedded)    │     │ (OpenAI) │
└────────────────────────┘     └──────────┘
┌────────────────────────┐     ┌──────────┐
│  Support Service         │────▶│ Provider │
│  (Spring AI embedded)    │     │ (OpenAI) │
└────────────────────────┘     └──────────┘

Consideration	Dedicated AI Service	Embedded Per-Service
Provider key/cost management	Centralized — one place to rotate keys, track spend, apply org-wide rate limits	Fragmented — each service manages its own, harder to get an org-wide cost picture
Advisor/prompt reuse	Centralized — shared Advisors (safety, logging, memory patterns) maintained once	Duplicated — each service reimplements or copy-pastes common Advisor logic
Latency	Extra network hop (service → AI service → provider)	Direct (service → provider)
Team ownership	Requires a dedicated team/on-call for the AI service, or it becomes a shared-ownership bottleneck	Each domain team owns their own AI integration, no cross-team dependency for changes
Blast radius of a provider outage	Contained to one service others degrade gracefully around	Same blast radius, but distributed — harder to get a single org-wide view of “is AI down right now”
Fit	Large orgs with many services needing AI, wanting centralized governance/cost control	Small-to-medium orgs, or a small number of services with really different AI needs that don’t share much

Common production compromise: a shared library (an internal Spring Boot starter wrapping common Advisors, observability conventions, and provider configuration defaults) embedded into each domain service, rather than either extreme — this gets centralized governance/consistency without the extra network hop and single-point-of-ownership bottleneck of a dedicated gateway service, at the cost of needing a versioning/distribution strategy for the shared library itself (standard internal-library governance, not AI-specific).

---

17.2 API Gateway Integration

Standard gateway patterns (Spring Cloud Gateway, Kong, an enterprise API management platform) apply to AI endpoints with AI-specific additions:

# Spring Cloud Gateway route example
spring:
  cloud:
    gateway:
      routes:
        - id: ai-chat-service
          uri: lb://ai-chat-service
          predicates:
            - Path=/api/chat/**
          filters:
            - name: RequestRateLimiter
              args:
                redis-rate-limiter.replenishRate: 5 # AI endpoints typically
                redis-rate-limiter.burstCapacity:
                  10 # rate-limited more
                  # conservatively than
                  # typical CRUD routes
                  # given cost-per-request
            - name: CircuitBreaker
              args:
                name: aiServiceCircuitBreaker
                fallbackUri: forward:/fallback/ai-unavailable

The circuit breaker here is really important beyond generic resilience — it’s the gateway-level enforcement of exactly the fallback/degradation patterns discussed at the service level in Section 4, ensuring that even if an individual AI service’s internal fallback chain (Section 4) is somehow exhausted, the gateway itself prevents a cascading failure from an unresponsive AI dependency taking down upstream callers.

---

17.3 Kafka / Event-Driven AI Processing

Not every AI interaction needs to be synchronous request/response. For workloads where the caller doesn’t need an immediate answer — bulk document classification, async report generation, background enrichment — an event-driven architecture decouples request submission from AI processing:

@Component
public class DocumentClassificationConsumer {

    private final ChatClient chatClient;
    private final KafkaTemplate<String, ClassificationResult> resultProducer;

    @KafkaListener(topics = "document-classification-requests", groupId = "ai-classifier")
    public void handle(DocumentClassificationRequest request) {
        try {
            ClassificationResult result = chatClient.prompt()
                    .user(request.documentText())
                    .call()
                    .entity(ClassificationResult.class);

            resultProducer.send("document-classification-results",
                    request.documentId(), result);
        } catch (Exception e) {
            // dead-letter handling — AI processing failures need the
            // SAME dead-letter-queue discipline as any other async
            // consumer failure; a failed classification shouldn't
            // silently vanish
            resultProducer.send("document-classification-dlq",
                    request.documentId(), errorResult(e));
        }
    }
}

Why this pattern matters specifically for AI workloads: it naturally absorbs the latency variability inherent to model calls (Section 16’s point about multi-second, variable-duration inference) without holding synchronous HTTP connections open, and it provides natural backpressure — if AI processing falls behind, Kafka’s consumer lag metric becomes your queue depth signal, and you scale consumer instances horizontally rather than the upstream producer blocking or timing out.

---

17.4 Async Processing Patterns

For synchronous-feeling UX over an actually-async backend (submit a request, poll or get notified when the AI result is ready) — a standard async-job pattern, not AI-specific in mechanism, but common in AI contexts given generation latency:

@PostMapping("/analysis-jobs")
public ResponseEntity<JobAccepted> submitAnalysis(@RequestBody AnalysisRequest request) {
    String jobId = UUID.randomUUID().toString();
    jobRepository.save(new Job(jobId, JobStatus.PENDING));
    kafkaTemplate.send("analysis-requests", jobId, request);
    return ResponseEntity.accepted().body(new JobAccepted(jobId, "/analysis-jobs/" + jobId));
}

@GetMapping("/analysis-jobs/{jobId}")
public ResponseEntity<JobStatusResponse> getStatus(@PathVariable String jobId) {
    Job job = jobRepository.findById(jobId).orElseThrow();
    return ResponseEntity.ok(new JobStatusResponse(job.status(), job.result()));
}

---

17.5 Distributed Systems Concerns Specific to AI

1. Idempotency: if a Kafka consumer processing an AI classification request crashes after calling the model but before committing the offset, a redelivery will call the model again — wasting real API cost, distinct from a typical idempotent database write’s near-zero-cost retry. Design AI-processing consumers with explicit dedup (check-before-process against a results store keyed by request ID) rather than assuming at-least-once semantics are cost-neutral to retry.
2. Distributed tracing across async boundaries: Section 13’s tracing concerns extend directly into Kafka-based flows — trace context propagation through message headers (standard practice for any Kafka-based distributed trace, applying identically here) is what makes a “submit → AI processes → result published” flow debuggable as one connected trace rather than three disconnected fragments.
3. Consistency between AI state and business state: a ChatMemory-persisted conversation and an order’s actual state in your order-service database are two different systems of record that can drift — design explicit reconciliation or accept eventual consistency deliberately, don’t assume they stay silently in sync.

---

17.6 Scalability

Horizontal scaling of AI-backed services follows standard stateless-service scaling principles provided the state-management guidance from earlier sections is actually followed: ChatMemory backed by shared storage (Section 8), rate limiting/token budgets backed by distributed counters (Section 15), no in-process caching of anything that needs cross-instance consistency. Given that discipline, scaling out is standard Kubernetes HPA/similar tooling — the AI-specific scaling signal worth adding beyond CPU/memory is provider rate-limit headroom (Section 4/13) as an autoscaling input, since a service can be well within CPU/memory limits while still being effectively saturated from a provider-rate-limit perspective, a signal generic infrastructure metrics won’t surface.

---

17.7 Fault Tolerance — Consolidated View

This section pulls together fault-tolerance guidance scattered across the series into one enterprise-level view:

Failure mode	Mitigation	Where covered
Provider outage/degradation	Multi-provider fallback chain	Section 4
Rate limiting	Proactive backoff using rate-limit metadata, distributed token budgets	Sections 4, 15
Transient network errors	RetryTemplate with exponential backoff	Section 3
Malformed/unparseable model output	Multi-tier structured-output error recovery	Section 11
Tool execution failure	Structured error results, ToolExecutionExceptionProcessor	Section 9
Cascading failure from AI dependency	Gateway-level circuit breaker	§17.2
Async processing failure	Dead-letter queue with explicit handling	§17.3
Multi-instance state inconsistency	Shared-storage memory/rate-limiting, never in-process state	Sections 8, 15, §17.6

The enterprise-architecture-level insight tying this together: none of these are solved by a single Spring AI feature — they’re solved by applying ordinary distributed-systems fault-tolerance discipline (the same discipline you’d apply to any external dependency) specifically and deliberately to the AI-provider dependency, which is often more failure-prone and more latency-variable than a typical internal microservice dependency, and therefore deserves at least as much fault-tolerance engineering, not less just because it’s “just an API call.”

---

17.8 Common Mistakes

1. No clear ownership model for a shared “AI service” that many teams depend on, causing it to become an under-resourced bottleneck.
2. Synchronous request/response for really async-appropriate workloads (bulk classification, report generation), holding connections/threads open unnecessarily instead of adopting an event-driven pattern.
3. Assuming Kafka’s at-least-once delivery is cost-neutral to retry for AI-processing consumers, without explicit idempotency/dedup design.
4. No distributed trace propagation across async AI-processing boundaries, making a multi-hop AI workflow undebuggable as one connected flow.
5. Scaling AI-backed services purely on CPU/memory metrics, missing provider-rate-limit headroom as a meaningful, AI-specific autoscaling signal.
6. Treating the AI provider dependency as inherently less deserving of fault-tolerance engineering than internal service dependencies, when it’s often more failure-prone and latency-variable.

---

17.9 Debugging

For “the AI feature is degraded but the AI service itself looks healthy” incidents, check provider-side status pages and rate-limit-headroom metrics (Section 13) before assuming an application-level bug — a meaningful fraction of production AI incidents are provider-side degradation, not application defects, and the gateway-level circuit breaker (§17.2) combined with the fallback chain (Section 4) should already be absorbing the worst of this if configured correctly; an incident review that finds neither engaged is itself a finding worth acting on.

---

17.10 Interview Questions

1. Compare dedicated AI-gateway-service and embedded-per-domain-service architectures, and describe the shared-library compromise pattern.
2. What AI-specific rate-limiting consideration should a gateway apply beyond typical CRUD-route rate limiting?
3. Why does Kafka’s at-least-once delivery semantics require different idempotency handling for AI-processing consumers than for typical database-writing consumers?
4. How would you propagate distributed trace context through a Kafka-based “submit → AI processes → result published” flow?
5. What autoscaling signal, specific to AI-backed services, goes beyond typical CPU/memory metrics, and why does it matter?
6. Describe the consolidated fault-tolerance table’s core insight — why isn’t AI-specific resilience solved by one Spring AI feature?
7. Why might an AI dependency deserve at least as much fault-tolerance engineering as an internal microservice dependency, arguably more?
8. What’s the architectural role of a gateway-level circuit breaker relative to a service-level provider-fallback chain (Section 4) — are they redundant?
9. How would you design an async job-status API for a long-running AI analysis task, and what HTTP semantics (status codes, polling vs. webhook) would you use?
10. What consistency risk exists between ChatMemory-persisted conversation state and a separate business system’s state, and how would you address it?
11. Why does an event-driven architecture naturally absorb AI inference latency variability better than synchronous request/response for appropriate workloads?
12. What ownership/team-structure risk does a shared dedicated AI service introduce, and how would you mitigate it organizationally?
13. Describe how you’d design dead-letter-queue handling for a failed AI classification consumer.
14. Why is a shared internal Spring Boot starter library often a practical middle ground between a dedicated AI gateway service and fully embedded per-service AI integration?
15. What’s the debugging first-step for “the AI feature is degraded but the service itself looks healthy,” and why?
16. How would you validate that your gateway-level circuit breaker and service-level fallback chain are actually engaging correctly during a provider outage, rather than assuming they work?
17. What’s the cost implication of a naive Kafka consumer retry on an AI-processing message without deduplication?
18. Why does centralizing provider API key management matter more at enterprise scale than at small scale?
19. How would you decide whether a given AI-backed workload belongs on a synchronous request/response path versus an async event-driven path?
20. What does “blast radius” mean in the context of comparing dedicated-AI-service versus embedded-per-service architectures during a provider outage?

---

17.11 Best Practices Checklist

• Choose dedicated-service, embedded, or shared-library architecture deliberately based on org size and cross-team AI usage, not by default.
• Apply gateway-level circuit breakers and rate limiting specifically tuned for AI endpoint cost/latency profiles.
• Use event-driven/async patterns for AI workloads that don’t require synchronous responses.
• Design explicit idempotency/dedup for any at-least-once AI-processing consumer.
• Propagate distributed trace context across async AI-processing boundaries.
• Add provider-rate-limit headroom as an autoscaling signal alongside CPU/memory.
• Apply the same fault-tolerance rigor to AI provider dependencies as to any other external dependency — arguably more, given typical latency/failure variability.

---

17.12 Key Takeaways

• Where AI capability lives architecturally (dedicated service, embedded, shared library) is a real design decision with concrete trade-offs in cost governance, latency, ownership, and blast radius — not a detail to default without thought.
• Event-driven/async patterns are a natural fit for AI’s inherent latency variability, decoupling submission from processing and providing natural backpressure via consumer lag.
• AI-processing consumers need idempotency design specifically because retrying a model call is meaningfully costly, unlike retrying a typical idempotent database write.
• Fault tolerance for AI dependencies is an application of ordinary distributed-systems discipline, deliberately and consistently applied — not a single framework feature to enable.
• This closes the 17-section core curriculum; the worked applications that follow show these architectural patterns assembled into complete, progressively harder, production-grade systems.

---

End of Section 17 — end of the core 17-section curriculum. Next: AI Chatbot with ChatClient (Spring Boot, Spring AI, conversation memory, prompt templates, streaming, Docker, JUnit) — the first of three complete, progressively harder production applications this series builds in full.