Evaluating, Observing and Controlling Agents

If an agent solves one test task, you know it worked once. You do not yet know whether it chooses good tools, wastes steps, fails safely, or succeeds across many different cases. Evaluation measures quality; observability shows what happened inside the run.

Test task → agent trajectory → final result
              ↓                  ↓
        trace each step      score outcome
              └──── diagnose and improve ────┘

What You Will Learn

• Why agent evaluation must inspect both the final result and the path taken.
• What to log for prompts, model decisions, tool calls, state changes, costs, and errors.
• How traces, datasets, metrics, human review, and model-based graders work together.
• How to test task success, tool choice, safety, latency, cost, and recovery behavior.
• How offline evaluation, production monitoring, and regression tests serve different purposes.

What a Real Agent Evaluation Records

Anthropic’s agent-evaluation guidance describes multi-turn tests in which an agent receives a task, tools, and an environment, then performs many model and tool calls while changing that environment. The complete message and tool-call trajectory becomes evidence for diagnosing the final result. (Anthropic, Demystifying Evals for AI Agents)

That is why “Did the final answer look correct?” is too small a test. A useful evaluation also asks whether the agent chose allowed tools, used sensible arguments, avoided waste, recovered safely, and left the environment in the expected state.

You’ve now spent eleven modules learning to build, secure, and reason about agents. This one asks the question that determines whether any of that worked: how do you know? Not “it answered my test question correctly when I tried it this morning” —, reliably know, across the volume and variety of real production traffic.

RAND reports that, by some estimates, more than 80% of AI projects fail, while Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027. The sources identify several causes—including misunderstood problems, poor data, inadequate infrastructure, escalating costs, unclear value, and weak risk controls—so these numbers must not be presented as proof of one single cause. (RAND research report, Gartner press release)

Evaluation and observability help with part of that larger problem: they reveal whether the system works, how it fails, and whether a change improves the measured outcome before failures reach more users.

Why “it worked once” isn’t evidence

Go back to Module 2’s founding distinction: traditional software is deterministic — same input, same output, every time — while an agent’s behavior is probabilistic. This has a direct, practical consequence for testing that’s worth stating precisely: when you run our support agent against a test case and it correctly diagnoses the payment issue, that single successful run tells you less than the equivalent test passing for traditional code would.

The same input, run again, could produce a different reasoning path — maybe still correct, maybe not. A single pass is a data point, not proof of reliability. This is the entire reason this module exists as a separate, dedicated subject rather than folding into “just write more tests.”

Why agents are harder to test than deterministic software

It’s worth being precise about why, not just asserting that it’s true. Traditional software testing works by asserting an exact, known-correct output for a given input — assert add(2, 2) == 4. This breaks down for agent output in two distinct ways.

First, there are often many valid, differently-worded ways to correctly resolve a task, so an exact-match assertion fails even when the agent did the right thing — Module 2 already established this for a single LLM response, and it’s equally true for a multi-step trajectory. Second, and more specific to agents: correctness isn’t only about the final answer.

An agent could reach the right final action through a bad process — unnecessary tool calls, a shaky diagnosis that happened to land on the right conclusion anyway — or reach a wrong final answer despite sound reasoning at every individual step, the way Module 4’s duplicate-authorization scenario showed is possible. Evaluating only the final output misses both of these real, distinct failure patterns.

The observability half: what you need to capture

Before you can evaluate anything, you need a complete record of what happened — this is the direct, practical foundation underneath everything else in this module, and it’s also directly what makes Module 10’s diagnostic process possible at all in a real production incident rather than only in a controlled walkthrough.

Logging captures individual, discrete events — a specific tool called with specific arguments, a specific result returned. Tracing is the complete, connected sequence of these events for one specific run, letting you reconstruct exactly what an agent did and why, after the fact. Tool-call visibility means specifically that every tool invocation — not just the final response — is recorded, since Module 10’s entire diagnostic method depends on being able to see which specific step in a trace went wrong, not just that the outcome was bad.

You’ve already seen what real, production-grade tracing looks like.

When Module 10’s Gemini CLI user filed his incident report, he included the “full powershell conversation,” and separately, the user who found Claude Code’s permission-bypass bug filed his report with “conversation JSONL, two debug logs, command history, and a written incident report” — and pointedly noted a gap in what had been logged: the system recorded a command’s output but not the command itself, which meaningfully limited how thoroughly the incident could be investigated.

That specific gap is worth sitting with, because it’s a direct, real illustration of this module’s point: observability isn’t automatic just because a system produces some logs — it requires deliberately capturing the specific things you’ll need when something eventually goes wrong, decided in advance, not reconstructed after the fact from whatever happened to be recorded.

The metrics that matter

Once you’re capturing traces, a real set of metrics becomes measurable, and each one tells you something different.

Latency is how long a task takes — both end-to-end and, more usefully, broken down per stage, so you can tell whether a slow run is the model itself, a specific tool, or steps running sequentially that could run in parallel. Token usage and cost are what a task consumes and costs — tracked per run, not just as a monthly aggregate, so a specific expensive pattern (Module 10’s infinite-loop failure, say) is attributable rather than lost in a total.

Task completion rate is the most fundamental outcome metric: across a representative set of real scenarios, how often does the agent achieve the stated goal — not “produce a plausible-looking response,” but satisfy what Module 3 defined as done. Success rate and failure rate are the same measurement from opposite sides, and it’s worth tracking failure rate specifically broken down by which of Module 10’s failure categories caused it, rather than one undifferentiated number.

Tool failure rate isolates how often a specific tool errors out, which often points to an integration problem worth fixing directly rather than an agent reasoning problem. Retry rate is a useful proxy for where reliability is weakest — a tool or a class of task with a consistently high retry rate is telling you something specific worth investigating, even before it becomes an outright failure.

Escalation rate — directly Module 9’s subject — tracks how often the agent hands off to a human, and whether that rate is trending in a healthy direction as the system improves, or a concerning one if it’s climbing unexpectedly.

The evaluation half: measuring correctness, not just activity

Metrics tell you what an agent did. Evaluation tells you whether what it did was right — and this is where the exact-match testing problem from earlier in this module needs a replacement, not a workaround.

A test scenario, for agent evaluation, is a realistic situation with a, checkable definition of success — not necessarily an exact expected output, but a rubric or a verifiable outcome. An evaluation dataset is a curated collection of these scenarios, representative of real usage, ideally including tricky edge cases deliberately — Module 4’s duplicate-authorization trap is exactly the kind of scenario worth including deliberately, precisely because it’s the failure category hardest to catch by accident.

SWE-bench: what a real evaluation dataset looks like

It’s worth seeing this principle implemented for real, at scale, because SWE-bench is about as clean a working example as exists of everything this module has been describing. It’s a benchmark built from real, closed GitHub issues and their real, merged pull-request fixes — an agent is given the actual issue description and the actual codebase, and has to produce a fix. Crucially, “success” here isn’t judged by whether the generated code looks correct — it’s determined by whether the repository’s own, real, pre-existing test suite passes afterward, the same objective, checkable standard a human engineer’s pull request would be held to.

This single design choice solves both of this module’s exact-match problems at once. There’s no single “correct” fix an agent’s output has to match word for word — many different code changes could make the tests pass, so exact-match scoring was never the right tool here in the first place.

And it evaluates the complete trajectory, not just a plausible-looking single step — an agent that generates elegant-looking code that doesn’t fix the bug fails, regardless of how convincing any individual step looked along the way.

This is precisely why Module 4 already referenced SWE-bench as evidence that “the loop, run correctly end to end, is the actual product” — now you can see exactly why that’s true: the benchmark’s entire design is built around measuring the end-to-end outcome against a, objective standard, not a subjective judgment of individual steps.

Evaluating a specific failure mode: InjecAgent

It’s worth knowing that this same evaluation discipline applies to security specifically, not just general task correctness — directly connecting this module back to Module 11. Researchers built InjecAgent, a benchmark specifically measuring how often an agent falls for indirect prompt injection when operating in a ReAct-style loop. Their finding is worth knowing precisely: even GPT-4, in a ReAct framework, was vulnerable roughly 24 to 47% of the time, depending on the specific scenario tested.

This is exactly Module 12’s principle applied to Module 11’s subject — you don’t just design guardrails and trust they work, you build a real evaluation dataset of attack scenarios and measure the actual rate at which they’re defeated, the same discipline SWE-bench applies to task correctness, now applied to injection resistance specifically.

Regression testing and model changes

Module 10 already told you that a model swap is a reliability risk, not a routine update — this module gives you the concrete mechanism for catching it before it reaches production. Regression testing, for an agent, means running your evaluation dataset — the same one, consistently — against a proposed change, and comparing the resulting scores against your current baseline, rather than checking for exact output equality.

A prompt change, a new tool, or a different underlying model should all be treated as changes that require re-running this same evaluation suite, precisely because behavioral evaluation — assessing how the agent behaves across a representative range of scenarios — is the only way to catch a regression that a handful of manual spot-checks would likely miss entirely.

This is directly why “it worked when I tried it after the model upgrade” is exactly as weak a claim as “it worked once” was at the start of this module — a few manual checks sample an infinitesimal fraction of the real behavior space a full evaluation dataset covers.

The current industry stack, and how it maps onto this module

It’s worth knowing what real teams are reaching for in production today, because the landscape has matured into recognizable categories that map directly onto this module’s own observability-versus-evaluation split — the industry organized itself around exactly the same distinction this module has been teaching.

AI-native trace and evaluation platforms treat the full agent trajectory — not a single call — as the primary object, capturing nested spans across reasoning steps, retrievals, and tool calls, then attaching evaluation scores directly to that captured trace. LangSmith, Langfuse, Arize Phoenix, and Braintrust are the names you’ll encounter most often here.

It’s worth knowing they specialize rather than competing on identical ground: LangSmith is purpose-built for LangChain and LangGraph specifically — you’ll meet both properly later in this learning path — with deep, framework-native integration. Langfuse and Arize Phoenix are open-source and self-hostable, a real, concrete answer for teams with data-residency or infrastructure-ownership requirements.

Braintrust leans deliberately evaluation-first, built around wiring regression checks directly into CI/CD — a real, working implementation of exactly this module’s “a model change should trigger a full evaluation run” principle, enforced automatically rather than left to a team remembering to do it.

Open-source evaluation libraries — RAGAS, DeepEval, MLflow’s evaluation tooling — focus specifically on the scoring layer: computing faithfulness, relevance, and task-completion scores, frequently using an LLM as the judge, often independent of any specific tracing platform.

AI gateways, like Helicone and Portkey, take a different approach: they sit as a proxy between your application and the model provider, capturing logging, cost tracking, and caching with minimal code changes — a real trade-off, since a proxy sees every individual call but, by design, doesn’t naturally capture the full orchestration graph the way an SDK-based tracer does.

And APM extensions — Datadog’s LLM Observability, Honeycomb, New Relic — bolt agent-specific tracing onto infrastructure monitoring teams already run, so AI-specific signals correlate directly with the same CPU, memory, and network metrics they’re already watching.

One important architectural detail worth knowing: much of this landscape is converging on OpenTelemetry, the same open standard traditional software observability already runs on, extended with gen_ai.* semantic conventions specifically for AI systems. Instrumenting against this standard once, rather than a specific vendor’s proprietary format, is what lets a team swap observability backends later without re-instrumenting their entire codebase — a practical answer to vendor lock-in, and worth knowing about even if you don’t reach for it on day one.

None of this changes what this module has taught you — every one of these tools exists specifically to capture the traces and compute the metrics already covered above. Which specific one a team reaches for in reality depends on their existing stack (already using LangGraph, already on Datadog), their infrastructure requirements (self-hosted versus managed), and how central automated regression gates are to their deployment process — a real, team-specific decision, not a universal right answer, and one worth making only once you understand what these platforms are built to capture, which is precisely what the rest of this module covered.

Applying this to our recurring agent

It’s worth making this concrete against the support agent one final time. A real evaluation setup for it would include a dataset of realistic tickets — routine ones, ambiguous ones, and deliberately tricky ones like Module 4’s mismatch-between-claim-and-data scenario and Module 4’s duplicate-authorization trap — each with a defined, checkable correct outcome, not necessarily an exact expected response.

Observability would capture every tool call in every run: which customer was checked, what payment history came back, what the gateway status check showed, and what final action was taken. Metrics tracked per run would include task completion rate against the dataset’s defined outcomes, tool failure rate specifically for check_payment_gateway (a external dependency worth watching separately), and escalation rate as a direct signal of how often the agent is correctly recognizing situations that exceed Module 9’s approval threshold rather than either overreaching or under-escalating.

Any change — a new prompt version, a different model, an added tool — would re-run against this same dataset before deployment, with a regression in any of these scores treated exactly as seriously as a failed test would be for traditional code.

What capturing this looks like in code

It’s worth making the observability half of this module concrete, extending exactly the tool-execution pattern from Module 5 and 9 one final time. A real trace record, built into the same handle_tool_call function you’ve now seen evolve across three modules, looks like this:

def handle_tool_call(tool_call, tool_registry, trace):
    start_time = time.time()
    entry = {"tool": tool_call.name, "arguments": tool_call.arguments}

    tool = tool_registry.get(tool_call.name)
    if tool is None:
        entry.update(status="error", error="unknown_tool")
        trace.record(entry)
        return error_result(f"Unknown tool: {tool_call.name}")

    try:
        result = tool.execute(**tool_call.arguments)
        entry.update(status="success", result=result,
                      latency_ms=round((time.time() - start_time) * 1000))
        trace.record(entry)
        return success_result(result)
    except Exception as e:
        entry.update(status="error", error=str(e),
                      latency_ms=round((time.time() - start_time) * 1000))
        trace.record(entry)
        return error_result(str(e))

Notice this records both the command and its result — directly the specific gap this module already flagged in a real incident report. Every field here maps to a metric from earlier in this module: latency_ms feeds latency tracking, status feeds tool failure rate, and the full sequence of recorded entries across one run is precisely what a trace is — not a separate system bolted on afterward, but a, deliberate byproduct of the same execution path every tool call already runs through.

Numbered Walkthrough: Score the Result and the Path

Assume an agent is tested on 100 support cases.

1. It resolves 82 cases correctly, so task success is 82%.
2. It chooses the correct tool in 90 cases, so tool-selection accuracy is 90%.
3. It violates a safety rule in 2 cases, so the safety-violation rate is 2%—an important problem even though overall success looks high.
4. Successful runs use an average of 4 tool calls, while failed runs use 11. This suggests some failures become wasteful loops.
5. The p95 latency is 18 seconds, meaning 95 of 100 runs finish within 18 seconds while the slowest 5 take longer.
6. Engineers inspect traces for those slow and unsafe runs, fix the responsible behavior, and rerun the same evaluation dataset before release.

One percentage cannot describe an agent. Outcome, trajectory, safety, cost, and latency answer different questions.

Common Misconception

Incorrect idea: A realistic final answer proves that the agent worked correctly.

Why it is incorrect: The agent may have used the wrong customer record, called unnecessary tools, ignored a failed action, or invented success. Evaluation must inspect the final environment state and the trajectory that produced it.

Key Takeaways

• A single successful run is a data point, not proof of reliability — agent behavior is probabilistic, so the same input can produce a different result on a different attempt.
• Traditional exact-match testing fails for agents in two distinct ways: many valid outputs can satisfy the same task, and correctness depends on the whole trajectory, not just the final answer.
• Observability requires deliberately capturing what you’ll need to diagnose a failure later — a real, documented incident showed a gap between logging a command’s output and logging the command itself, meaningfully limiting the investigation.
• Metrics like latency, cost, task completion rate, tool failure rate, retry rate, and escalation rate each measure something different, and should be tracked per-run, not just as aggregates.
• SWE-bench is a real, working example of a well-designed evaluation dataset — judging success by an objective, checkable standard (does the real test suite pass) rather than subjective similarity to one “correct” answer.
• InjecAgent applies this same evaluation discipline to security specifically, finding GPT-4 in a ReAct framework vulnerable to indirect injection roughly 24-47% of the time — proof that guardrails need to be measured, not just designed and trusted.
• A model swap, a prompt change, or a new tool should always trigger a full regression run against your evaluation dataset — a few manual spot-checks sample too little of the real behavior space to catch what matters.

Think Like an AI Engineer

• Design three deliberately tricky scenarios for our support agent’s evaluation dataset, beyond the two already mentioned in this module. For each one, define what “success” means in a way that doesn’t require an exact-match output.

• A teammate proposes measuring agent quality with a single number: overall task completion rate. Using this module’s full metric list, what failure pattern could this one number completely miss?

• Go back to the Claude Code incident report mentioned in this module — logging a command’s output but not the command itself. Pick a tool from our support agent’s toolset and specify exactly what you would log for every call to it, and justify why each field matters for a future investigation.

• InjecAgent found GPT-4 vulnerable to indirect injection 24-47% of the time depending on the scenario. If you were responsible for an agent with real production traffic, what would you do with a number like that — and how would you decide whether your own specific guardrails (Module 11) had moved that number for your system?

Module 13 turns from principle to practice: real, documented examples of how companies across different industries are using agentic AI in production today — what’s working, what isn’t, and the concrete lessons each one offers.