TechByteByByte

AI Reliability

Closing Level 5: why AI systems fail — model/provider/network failures, timeouts, hallucinations — and the reliability patterns (retries, circuit breakers, fallbacks, graceful degradation) that keep a system functioning when a dependency inevitably breaks.

#AI Engineering#Reliability#Level 5

Begin with the problem

External AI dependencies will time out, throttle, return malformed data, or disappear. Reliability engineering decides how the system fails without lying, looping forever, or taking down unrelated features.

dependency call → timeout/retry/backoff → circuit breaker → fallback/degrade → recover

What you will learn

  • Use bounded retries, timeouts, backoff, circuit breakers, and fallbacks.
  • Separate graceful degradation from fabricated success.
  • Choose recovery behavior based on idempotency and user impact.

Current production grounding: Kubernetes documents workload autoscaling and controlled Deployments for operating containerized services.

These links document current public features and practices. They do not reveal a provider’s private implementation or guarantee that every product uses identical defaults.

Topic-specific reference: Kubernetes describes self-healing and controlled rollouts for resilient distributed workloads.

1. The Engineering Problem

Module 1 framed the model as an external dependency. Every external dependency, without exception, eventually fails — a provider has an outage, a network call times out, a rate limit gets hit. This module covers the reliability patterns that keep your system functioning through those failures, rather than going down entirely the moment one dependency does.

2. Why AI Systems Fail — The Failure Sources

Failure SourceWhat Happens
Model provider outageThe provider’s API is unavailable — entirely outside your control
Network failuresRequests time out or drop before reaching the provider
Rate limitsThe provider rejects requests once you exceed a quota
Malformed outputThe model returns output that fails structured-output validation (Module 9)
Retrieval/vector DB failuresYour RAG system’s infrastructure dependency goes down
HallucinationThe model produces a plausible but ungrounded answer — a quality failure, not an availability one

Some of these are availability failures (the dependency is down); others are quality failures (the dependency responded, but incorrectly). Reliability engineering addresses both — Module 10’s evaluation catches quality failures; this module’s patterns address availability failures.

3. Retries With Exponential Backoff

A transient failure (a momentary network blip) often
succeeds on retry. But retrying IMMEDIATELY and REPEATEDLY can
make things worse -- hammering an already-struggling
provider.

EXPONENTIAL BACKOFF:      wait progressively LONGER between retries
                         (1s, 2s, 4s, 8s...) -- giving the
                         dependency time to recover, rather
                         than adding to its load.

4. Timeouts — A Non-Negotiable Requirement

WITHOUT a timeout: a single slow model call can hang a
                   request indefinitely, consuming a connection and
                   blocking a user with no response at all.

WITH a timeout: the request fails FAST, allowing a
                        fallback strategy (Section 6) to take over.

5. Circuit Breakers — Stop Calling a Dependency That’s Down

Directly a software-reliability pattern, applied here to AI
dependencies specifically:

CLOSED:      healthy -- requests flow normally

OPEN:            after a FAILURE THRESHOLD is crossed, the circuit
                "opens" -- requests short-circuit
                IMMEDIATELY to a fallback, without even attempting
                the failing dependency

HALF-OPEN:            after a recovery timeout, the circuit
                     allows ONE test request through --
                     if it succeeds, close the circuit; if not,
                     stay open

A circuit breaker prevents your system from wasting time and resources repeatedly trying a dependency that’s already known to be down — every request during an outage would otherwise wait for a full timeout before failing.

6. Fallback Strategies — Graceful Degradation

FALLBACK MODEL:      if the primary model provider is down, route to a DIFFERENT provider or a
                    self-hosted model -- degraded quality, but
                    still functional

FALLBACK PROMPT:         a SIMPLER, more constrained prompt that's
                        more likely to succeed under
                        degraded conditions

GRACEFUL DEGRADATION:        the system reduces
                            functionality rather than failing
                            entirely -- e.g., "search is temporarily
                            unavailable, here's a direct link to our
                            FAQ page" instead of a complete outage

7. A Real-World Analogy — The Power Grid, Revisited

Module 11's power-grid analogy: a WELL-ENGINEERED grid
doesn't go COMPLETELY dark when ONE power plant fails -- it
REROUTES load to other sources (fallback), SHEDS
non-critical load DELIBERATELY to protect the core system (graceful
degradation), and STOPS drawing from a failed source
until it's confirmed recovered (circuit breaker).

An AI system deserves this SAME engineering discipline around its
model and infrastructure DEPENDENCIES.

8. Idempotency, Dead Letter Queues, and Bulkheads

IDEMPOTENCY:      directly your Agents course's Module 8 principle
                 -- a retried operation should NEVER cause a
                 duplicate real-world effect

DEAD LETTER QUEUE:      when a request fails after all
                       retries, route it to a separate queue for
                       later investigation, rather than SILENTLY
                       dropping it

BULKHEADS:                 isolate failures -- one
                          tenant's or feature's excessive load or
                          failure shouldn't be able to exhaust
                          resources needed by OTHER tenants/features

9. A worked developer example

TechCorp’s model-call reliability wrapper:

FailurePattern AppliedResult
Primary model provider downCircuit breaker opens after 3 failuresRequests short-circuit to a fallback model immediately
A single request’s network call dropsRetry with exponential backoffsucceeds on the 2nd attempt
Fallback model also failsGraceful degradationUser sees “search temporarily limited” rather than a hard error
A request fails after all retries and fallbacksDead letter queueLogged for investigation, rather than silently lost

10. How Is This Used in the Industry?

🤖 How Is This Used in the Industry?

Production AI systems wrap every external model or infrastructure call in this exact combination of patterns — timeout, bounded retry with backoff, circuit breaker, and a defined fallback — precisely because provider outages and transient failures are a recurring reality, not a rare edge case worth ignoring.

11. Common Mistakes

Incorrect idea: No timeout on model calls.

Why it is incorrect: As shown directly in Section 4, this risks requests hanging indefinitely.

Incorrect idea: Retrying without backoff, or without a bound.

Why it is incorrect: As shown directly in Section 3, this can worsen an already- struggling dependency’s load.

Incorrect idea: No fallback strategy at all — a single dependency failure takes down the entire system.

Why it is incorrect: As shown directly in Section 6-7, graceful degradation is achievable with deliberate design.

12. Code — A Circuit Breaker With Fallback

What this shows: a working circuit breaker wrapping a failing dependency — exactly Section 5’s state machine and Section 9’s worked developer example, made into runnable reliability infrastructure.

from enum import Enum
import time

class CircuitState(Enum):
    CLOSED = "closed"       # healthy, requests flow normally
    OPEN = "open"            # failing, requests short-circuit
    HALF_OPEN = "half_open"  # testing recovery

class CircuitBreaker:
    """A circuit breaker (Section 5) -- stops sending
    requests to a failing dependency after a threshold of failures,
    rather than letting every request wait for a timeout against a
    dependency that's already known to be down."""

    def __init__(self, failure_threshold: int = 3, recovery_timeout_s: float = 5.0):
        self.failure_threshold = failure_threshold
        self.recovery_timeout_s = recovery_timeout_s
        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.last_failure_time = None

    def call(self, fn, fallback_fn):
        if self.state == CircuitState.OPEN:
            if time.time() - self.last_failure_time >= self.recovery_timeout_s:
                self.state = CircuitState.HALF_OPEN
            else:
                return fallback_fn()  # short-circuit -- don't even TRY the failing dependency

        try:
            result = fn()
            if self.state == CircuitState.HALF_OPEN:
                self.state = CircuitState.CLOSED
                self.failure_count = 0
            return result
        except ConnectionError:
            self.failure_count += 1
            self.last_failure_time = time.time()
            if self.failure_count >= self.failure_threshold:
                self.state = CircuitState.OPEN
            return fallback_fn()

def failing_primary_model():
    raise ConnectionError("Primary model provider is down")

def fallback_model():
    return "Fallback model response (degraded but functional)"

breaker = CircuitBreaker(failure_threshold=3, recovery_timeout_s=100)

results = []
for i in range(5):
    result = breaker.call(failing_primary_model, fallback_model)
    results.append((i + 1, breaker.state.value, result))

for attempt, state, result in results:
    print(f"Attempt {attempt}: circuit={state}, result='{result}'")

Expected Output:

Attempt 1: circuit=closed, result='Fallback model response
(degraded but functional)'
Attempt 2: circuit=closed, result='Fallback model response
(degraded but functional)'
Attempt 3: circuit=open, result='Fallback model response (degraded
but functional)'
Attempt 4: circuit=open, result='Fallback model response (degraded
but functional)'
Attempt 5: circuit=open, result='Fallback model response (degraded
but functional)'

What this confirms: The circuit stays CLOSED for the first two failed attempts, so the system still tries the primary dependency. It OPENS when the third failure crosses the configured threshold.

Every later attempt goes directly to the fallback without calling the known-failing primary. This is Section 5’s state machine and Section 9’s worked example operating as designed.

13. Production Considerations

  • The recovery_timeout_s (how long to wait before testing recovery) should be tuned per dependency — too short wastes effort retrying a still-down service; too long delays recovery detection
  • Log every circuit state transition (Module 12) — a circuit opening is a real production signal worth alerting on

14. Trade-offs

  • Fallback models provide lower quality than the primary — a real, deliberate trade-off of quality for availability during an outage
  • Circuit breakers add state and complexity to every external call — worthwhile specifically for dependencies with a real, meaningful chance of failure

15. Chapter Summary

AI systems depend on external components — model providers, vector databases, networks — that will, eventually fail.

Reliability engineering combines timeouts (fail fast), bounded retries with backoff (recover from transient issues without worsening load), circuit breakers (stop calling a known-failing dependency), and fallback strategies (degrade gracefully rather than failing completely) into a layered defense — directly mirroring traditional software reliability engineering, applied specifically to AI’s external dependencies.

16. Visual Cheat Sheet

Timeout       -->  fail FAST, don't hang indefinitely
Retry+Backoff -->  recover from TRANSIENT failures, without piling on
Circuit Breaker --> stop calling a KNOWN-failing dependency
Fallback      -->  degrade GRACEFULLY, don't fail completely

17. Top Takeaways

  1. Every external AI dependency (model provider, vector DB) will, eventually fail — plan for it structurally.
  2. Timeouts are non-negotiable — a hung request with no timeout blocks users indefinitely.
  3. Retries need exponential backoff and a bound, not unlimited, immediate retrying.
  4. Circuit breakers stop wasted effort against a known- failing dependency, short-circuiting to a fallback immediately.
  5. Graceful degradation (reduced functionality) is better than complete failure when a dependency is down.

18. Interview Questions

Q: 1. Why is a circuit breaker more effective than retry logic alone during a provider outage?**

Ans: Retry logic alone still attempts the failing dependency on every single request, each one waiting for a timeout before failing — wasteful and slow during a sustained outage.

A circuit breaker tracks failure history and, once a threshold is crossed, short-circuits directly to a fallback without even attempting the known-failing dependency, saving time and resources until the dependency is confirmed to have recovered.

  • Why it matters: During a real outage, this difference determines whether users experience slow, hanging requests or an immediate, if degraded, response.
  • Real-world example: Section 12’s code — after the third failure, every subsequent attempt short-circuits immediately.
  • Common mistake: Implementing retries without a circuit breaker, so every request during a sustained outage still pays the full retry-and-timeout cost.
  • Interviewer is testing: Whether the candidate understands these as complementary, not redundant, patterns.
  • Likely follow-up: “How would you decide the failure threshold and recovery timeout?” → tuned to the dependency’s real failure characteristics and your system’s tolerance for degraded operation.

Q: 2. Design a fallback strategy for a RAG system when the vector database becomes unavailable.**

Ans: I’d first apply a timeout and circuit breaker around the vector DB call. On failure, rather than a hard error, I’d degrade gracefully — perhaps falling back to a cached response for common queries, or explicitly informing the user that search is temporarily limited while still allowing the system to function for requests that don’t need retrieval.

  • Why it matters: A single infrastructure dependency failing shouldn’t take down the entire user experience.
  • Real-world example: Section 9’s TechCorp table — “search temporarily limited” instead of a hard error.
  • Common mistake: Having no fallback at all, so a vector DB outage produces a complete, hard failure for every request.
  • Interviewer is testing: Whether the candidate can design a concrete degradation path rather than just naming the pattern abstractly.
  • Likely follow-up: “What would you log during this degraded state?” → Every fallback activation (Module 12), so the team knows how often and how long the system operated degraded.

19. Scenario-Based Question

Scenario: TechCorp’s model provider has a 20-minute outage. During this window, every user request takes the full 30-second timeout before failing, and the team discovers there was no circuit breaker or fallback model configured — every single request during the outage experienced this same slow failure.

  • Problem Analysis: Section 11’s common mistake — retries and timeouts existed, but no circuit breaker or fallback meant every request still paid the full failure cost individually.
  • How to Think: A 30-second wait before failure, repeated for every single request during a 20-minute outage, represents a poor user experience that a circuit breaker would have prevented after the first few failures.
  • Investigation: Confirm the current reliability wrapper only implements timeout and retry, with no circuit breaker or fallback model configured.
  • Root Cause: Missing circuit breaker (Section 5) and fallback strategy (Section 6) — reliability infrastructure was incomplete.
  • Solution: Implement Section 12’s circuit breaker pattern around the model call, paired with a fallback model or graceful degradation path, so that after an initial handful of failures, subsequent requests during the outage fail fast or succeed via fallback instead of each waiting the full timeout.
  • Trade-offs: A fallback model provides lower quality responses during the outage — an accepted, deliberate trade-off favoring availability over the primary model’s full quality during a real outage window.
  • Production Considerations: This scenario directly demonstrates why Section 15 treats these patterns as a necessary, layered combination — timeout and retry alone are insufficient without a circuit breaker and fallback completing the reliability picture.

20. Next Step

Next: Module 15 — AI Cost Engineering — Level 6 begins here: what makes AI applications expensive, and the concrete techniques (caching, model routing, batching) that control it.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed