Failure Engineering

Begin with the problem

Reliability code is only a promise until failure is deliberately triggered in a safe environment. Failure engineering verifies that the system degrades honestly and recovers as designed.

inject controlled failure → observe response → verify fallback/alert → repair gap → repeat

What you will learn

• Test provider, vector database, tool, state, security, cost, and loop failures.
• Define the expected user-visible and operator-visible response for each failure.
• Run failure experiments safely in tests or staging, never blindly in production.

Current production grounding: OpenAI’s Evals documentation shows dataset- and grader-based evaluation for model applications.

These links document current public features and practices. They do not reveal a provider’s private implementation or guarantee that every product uses identical defaults.

1. The Engineering Problem

Module 30 assembled the complete, correctly-functioning system. This module deliberately breaks it — 15+ realistic failure scenarios, each with the exact, correct system response, so you have a concrete playbook rather than hoping your reliability code (Module 14) works when a real incident actually happens.

---

2. Why Deliberately Breaking a System Is Valuable

Module 24's chaos-testing principle: HOPING reliability
code works is different from VERIFYING it does. Failure
engineering means DELIBERATELY inducing each scenario below (in a
test or staging environment) and confirming the system responds
EXACTLY as expected -- not discovering a gap during a real,
production incident.

---

3. The Complete Failure Playbook

#	Scenario	Expected System Response	Module
1	LLM provider unavailable	Circuit breaker opens after threshold; requests route to fallback model or degrade gracefully	14
2	Vector DB unavailable	Fall back to cached responses, or inform the user search is temporarily limited; system stays functional	14
3	Traditional database unavailable	degrade the specific feature depending on it; other features remain unaffected (bulkhead isolation)	14
4	Tool unavailable (agent context)	Retry with backoff; if exhausted, agent reports the specific tool failure rather than guessing at a result	8, 14
5	Model call times out	Request fails fast (timeout), triggers fallback rather than hanging indefinitely	14
6	Rate limit exceeded (provider)	Request queues or retries with backoff; system-level rate limiting prevents cascading further limit breaches	14, 17
7	Bad/irrelevant retrieval	Agentic RAG evaluates retrieval quality and reformulates the query, or informs the user no relevant information was found	7
8	Hallucination detected	Realtime groundedness check (Module 30’s architecture) blocks the response before it reaches the user	10, 30
9	Direct prompt injection	Input guardrail blocks the request before it ever reaches the model	13
10	Indirect prompt injection (via retrieved content)	Retrieved-content scan blocks the malicious instruction before it’s included in context	13
11	Malformed JSON output	Structured-output validation rejects it; retry with a specific error, then escalate to a human if still invalid	9
12	Agent enters an infinite loop	Max-iterations limit and cost cap trigger; agent halts and reports failure rather than looping forever	8
13	Cost explosion (single task)	Real-time cost tracking triggers an alert; per-task cost cap halts further spend on that specific task	15
14	Latency spike (specific stage)	Per-stage latency budget tracking (Module 16) identifies the specific slow stage for targeted investigation	16
15	Memory/context explosion	Token budget (Module 6, 15) caps context size; oversized content is truncated or rejected before the model call	6, 15
16	Tenant data cross-contamination attempt	Structural, data-layer tenant isolation blocks the query regardless of application-layer logic	13
17	Deployment regression (post-canary)	Automated rollback triggers on error-rate threshold breach; traffic reverts to the previous stable version	25
18	Golden dataset silently modified	Dataset versioning (Module 18, 27) makes the change visible and attributable in the registry’s audit trail

---

4. Two Detailed Walkthroughs

Scenario: LLM provider unavailable (#1)

TRIGGER (in testing): simulate the provider's endpoint returning
                      connection errors for every request.

EXPECTED, system behavior:

  1. First few requests: retry with exponential backoff (Module 14)
  2. After the failure threshold: circuit breaker OPENS (Module 14)
  3. Subsequent requests: SHORT-CIRCUIT immediately to a fallback
     model, WITHOUT even attempting the known-failing primary
  4. Observability (Module 12) logs every circuit-state transition
  5. Alerting fires on the circuit opening -- a real, actionable
     signal for the on-call engineer

VERIFICATION: confirm requests during the simulated outage
             complete via the fallback path within a
             REASONABLE, bounded time -- not each waiting a full
             timeout before failing.

Scenario: Agent enters an infinite loop (#12)

TRIGGER (in testing): construct a task where the agent's reasoning
                      cannot determine it has finished
                      (e.g., a tool that always returns an
                      ambiguous result).

EXPECTED, system behavior:

  1. Agent continues iterating, exactly as it would in a
     REAL stuck scenario
  2. Max-iterations limit (Module 8) is REACHED
  3. Agent HALTS and returns a failure/incomplete status --
     NOT a fabricated, guessed "success"
  4. Cost incurred is BOUNDED by the iteration limit, not unbounded
  5. Observability logs the FULL trace, enabling root-cause
     diagnosis of why the agent couldn't determine completion

VERIFICATION: confirm the agent stops at the configured
             limit, and that the reported failure is HONEST (not a
             fabricated success) and diagnosable from the logged
             trace.

---

5. A Real-World Analogy — The Power Grid, Once More

Module 14 and 25's power-grid analogy: a WELL-RUN grid
operator doesn't wait for a REAL blackout to discover whether their
failover systems work -- they run scheduled, deliberate
drills, disconnecting a power source in a CONTROLLED way and
verifying the grid reroutes correctly.

Failure engineering is EXACTLY this same discipline, applied to an
AI system's reliability, security, and cost-control mechanisms.

---

6. How Is This Used in the Industry?

🤖 How Is This Used in the Industry?

Mature AI engineering teams run scheduled chaos-testing exercises (Module 24) against a staging environment, deliberately inducing scenarios from Section 3’s playbook and verifying the system’s actual response matches the expected one — precisely so a real production incident is a rehearsed, well-understood event rather than a novel crisis.

---

7. Common Mistakes

Incorrect idea: Never actually testing failure scenarios, only assuming reliability code works.

Why it is incorrect: As shown directly in Section 2, this is the difference between hoping and verifying.

Incorrect idea: Treating a “graceful failure” as equivalent to a fabricated success.

Why it is incorrect: As shown directly in Section 4’s second walkthrough, an honestly-reported failure is correct behavior — silently guessing at a result is not.

Incorrect idea: Testing only the “happy path” reliability scenarios and skipping adversarial ones (injection, tenant isolation).

Why it is incorrect: As shown directly in Section 3’s rows 9-10 and 16, security failure scenarios deserve the same deliberate testing discipline.

---

8. Code — A Failure Scenario Playbook Lookup

What this shows: a working playbook that a real on-call engineer or automated system could query during an actual incident — directly implementing Section 3’s table as queryable, reference data, exactly what a real incident-response runbook looks like in practice.

from dataclasses import dataclass
from enum import Enum

class FailureScenario(Enum):
    LLM_UNAVAILABLE = "llm_provider_unavailable"
    AGENT_INFINITE_LOOP = "agent_infinite_loop"
    COST_EXPLOSION = "cost_explosion"

@dataclass
class FailureResponse:
    scenario: FailureScenario
    expected_response: str
    module_reference: str

# Section 3's playbook, made into structured, queryable data
FAILURE_PLAYBOOK = {
    FailureScenario.LLM_UNAVAILABLE: FailureResponse(
        FailureScenario.LLM_UNAVAILABLE,
        "Circuit breaker opens after threshold; requests route to fallback model or degrade gracefully",
        "Module 14",
    ),
    FailureScenario.AGENT_INFINITE_LOOP: FailureResponse(
        FailureScenario.AGENT_INFINITE_LOOP,
        "Max iterations and cost cap trigger; agent halts and reports failure rather than looping forever",
        "Module 8",
    ),
    FailureScenario.COST_EXPLOSION: FailureResponse(
        FailureScenario.COST_EXPLOSION,
        "Real-time cost tracking triggers an alert; per-task cost cap halts further spend on that task",
        "Module 15",
    ),
}

def get_expected_response(scenario: FailureScenario) -> FailureResponse:
    """Directly implements Section 8's runbook lookup -- exactly what
    an on-call engineer would query during a real incident, or what
    an automated alerting system would use to route the right
    playbook entry."""
    return FAILURE_PLAYBOOK.get(scenario)

for scenario in [FailureScenario.LLM_UNAVAILABLE, FailureScenario.AGENT_INFINITE_LOOP, FailureScenario.COST_EXPLOSION]:
    response = get_expected_response(scenario)
    print(f"[{response.scenario.value}]")
    print(f"  Expected: {response.expected_response}")
    print(f"  See: {response.module_reference}\n")

Expected Output:

[llm_provider_unavailable]
  Expected: Circuit breaker opens after threshold; requests route
to fallback model or degrade gracefully
  See: Module 14

[agent_infinite_loop]
  Expected: Max iterations and cost cap trigger; agent halts and
reports failure rather than looping forever
  See: Module 8

[cost_explosion]
  Expected: Real-time cost tracking triggers an alert; per-task
cost cap halts further spend on that task
  See: Module 15

What this confirms: each scenario correctly retrieves its expected, system response and originating module reference — exactly the kind of quick, reliable lookup a real on-call engineer needs during an actual incident, or that an automated alerting system could use to surface the correct runbook entry immediately rather than requiring a manual search.

---

9. Production Considerations

• schedule regular chaos-testing exercises against this playbook (Module 24) — a playbook that’s never actually tested against real system behavior provides false confidence
• Keep this playbook updated as new failure modes are discovered in real incidents — treat it as a living document, not a one-time deliverable

---

10. Trade-offs

• Deliberately inducing failures in a staging environment requires, dedicated testing infrastructure and time — worthwhile given the alternative is discovering gaps during a real incident
• A comprehensive playbook takes, ongoing maintenance as the system evolves — a stale playbook can itself be misleading during a real incident

---

11. Chapter Summary

Failure engineering means deliberately, systematically inducing realistic failure scenarios — provider outages, malformed output, infinite loops, cost explosions, security attacks — and verifying the system’s actual response matches the correct, expected one from this module’s playbook, rather than hoping the reliability patterns from Modules 8-17 work correctly when a real incident eventually occurs.

A graceful, honestly-reported failure (an agent halting and reporting incompletion) is correct behavior; a fabricated success is not. This discipline turns real production incidents into rehearsed, well-understood events rather than novel crises.

---

12. Visual Cheat Sheet

Availability failures (#1-6):   circuit breakers, fallbacks, retries
Quality failures (#7-8):        agentic RAG, groundedness checks
Security failures (#9-10, 16):  guardrails, structural isolation
Structural failures (#11-12):   validation gates, iteration limits
Resource failures (#13-15):     cost caps, latency budgets, token
                                 budgets
Operational failures (#17-18):  automated rollback, dataset
                                 versioning

---

13. Top Takeaways

1. Failure engineering means deliberately inducing realistic failures and verifying the actual response, not hoping reliability code works.
2. A graceful, honestly-reported failure is correct behavior — a fabricated success is a worse outcome than an honest failure.
3. Each failure scenario has an exact, expected response traceable to a specific module’s reliability, security, or resource-control pattern.
4. Chaos-testing exercises should be scheduled and repeated, not run once and forgotten.
5. This playbook is a living document — update it as new failure modes are discovered in real incidents.

---

14. Interview Questions

Q: 1. Why is a fabricated success worse than an honestly-reported failure when an agent cannot complete a task?**

Ans: A fabricated success gives downstream systems and users false confidence that a task completed correctly, which can propagate incorrect information or trigger further actions based on a false premise.

An honestly-reported failure — the agent halting at its max-iterations limit and reporting it couldn’t determine completion — is diagnosable and doesn’t mislead anyone downstream, even though it’s a less satisfying immediate outcome.

• Why it matters: This distinction directly shapes how failure- handling code should be designed — favoring honest failure signals over confident-sounding but incorrect output.
• Real-world example: Section 4’s second walkthrough — the agent reports failure rather than guessing at a result.
• Common mistake: Designing a system that always produces SOME response, even when that response is unreliable or fabricated.
• Interviewer is testing: Whether the candidate values honest, diagnosable failure over superficially “complete” but unreliable output.
• Likely follow-up: “How would you design an interface to communicate a failure gracefully to an end user?” → A clear, honest message about the limitation, potentially with a path to human escalation, rather than either a fabricated answer or a generic, unhelpful error.

Q: 2. Why should failure scenarios be tested deliberately in staging rather than only relying on reliability code being theoretically correct?**

Ans: Reliability code — circuit breakers, retries, fallbacks — can have bugs or misconfigurations that only become apparent when the actual failure condition occurs.

Deliberately inducing the failure in a staging environment and verifying the system’s response matches expectations catches these gaps before a real production incident does, turning what could be a novel crisis into a rehearsed, well-understood event.

• Why it matters: This is the core distinction between hoping and verifying reliability.
• Real-world example: Section 5’s power-grid drill analogy.
• Common mistake: Writing reliability code and considering the work complete without ever testing it against the actual failure condition it’s meant to handle.
• Interviewer is testing: Whether the candidate treats reliability as something to be verified, not just implemented and assumed correct.
• Likely follow-up: “How often would you run these chaos-testing exercises?” → on a regular schedule, and especially after any significant change to reliability-related code, rather than as a one-time validation.

---

15. Scenario-Based Question

Scenario: TechCorp has implemented circuit breakers, retries, and fallback models per Module 14, but has never actually tested these mechanisms against a real, simulated provider outage. During an actual outage, the team discovers the fallback model’s API credentials were never configured — the fallback path silently fails too, producing hard errors for every user during the entire outage window.

• Problem Analysis: Section 2 and 9’s point — reliability code was implemented but never verified against the actual failure condition it was designed to handle.
• How to Think: This gap was completely preventable — a single staging-environment chaos test (Section 8, #1) would have caught the missing fallback credentials before any real users were affected.
• Investigation: Confirm the fallback model configuration was never exercised in any test or staging environment before this real incident.
• Root Cause: No deliberate, scheduled chaos-testing practice (Section 2) — reliability code was implemented but never actually verified end-to-end.
• Solution: Fix the fallback credentials immediately; implement Section 8’s playbook as a scheduled chaos-testing practice going forward, specifically including inducing a real provider outage in staging and confirming the fallback path works end-to-end, not just that the code exists.
• Trade-offs: Building and maintaining chaos-testing infrastructure requires, ongoing investment — a real, worthwhile cost given this incident directly demonstrates the alternative.
• Production Considerations: This scenario directly demonstrates Section 2’s core point — reliability code that has never been tested against its actual failure condition provides false confidence, precisely the gap failure engineering exists to close.

---

16. Next Step

Next: Module 32 — The AI Engineering Career — Level 12 begins here: the junior-to-architect progression, what differentiates senior engineers, and what interviewers actually expect.