TechByteByByte

Observability

Why ordinary application logging isn't enough for agents, and what a real agent trace needs to capture — logging, metrics, tracing, trajectories, and tool calls — to make diagnosis actually possible in production.

#AI Agents#AI#Observability#Level 8

Begin with the problem

When an agent fails, teams need the complete trace: prompts, decisions, tool calls, results, state changes, retries, cost, timing, and stop reason.

agent event → logs + metrics + trace → diagnose trajectory → connect failure to evidence

What you will learn

  • Distinguish logs, metrics, traces, events, and agent trajectories.
  • Record decisions, tool calls, results, state changes, timing, cost, and stop reasons.
  • Protect sensitive prompt and tool data while retaining useful evidence.
  • Use trace evidence to debug failures and improve evaluations.

Current real-system grounding: OpenAI’s evaluation guidance supports dataset-based testing, and Google’s tools guide makes the application/tool execution boundary explicit.

These official links document available product features. They do not reveal every provider’s private implementation, hidden reasoning, default setting, or internal limit.

1. The problem this module solves

Module 19 gave you a failure taxonomy; Module 20 gave you evaluation metrics. Neither is actually usable in production without real visibility into what an agent actually did — this module covers exactly what that visibility requires.

2. Why Ordinary Application Logging Isn’t Enough

ORDINARY application logging: typically captures ERRORS and maybe
                              the FINAL request/response -- sufficient for most traditional
                              software (Module 1's evolution story).

An AGENT'S operation: involves MULTIPLE reasoning steps, tool calls,
                      and state transitions -- ALL necessary to understand WHY a specific final
                      answer was produced (directly Module 19's
                      diagnostic need).

If you only log the final answer, you can observe THAT something went wrong (Module 20’s evaluation), but NOT diagnose WHERE in the trajectory it went wrong (Module 19’s failure taxonomy) — exactly the same principle your RAG course established for retrieval pipelines, now applied to an agent’s full reasoning process.

3. What an AI Agent Trace Needs to Capture

ElementWhy It’s Needed
Decision summaries / action rationalesThe generated rationale, selected action, and evidence available at each step; this is not hidden internal reasoning
Tool callsWhich tool, with what arguments, and what result (Module 6-7)
State transitionsHow the agent’s understanding evolved (Module 12)
Latency per stepWhere real time is being spent
Token usage / costDirectly Module 20’s cost dimension, per step
Errors and retriesDirectly Module 19’s tool-failure and infinite-loop diagnostics

4. Logging, Metrics, and Tracing — Three Distinct

Concepts

LOGGING:      individual, discrete EVENTS ("tool X was called with
             arguments Y")

METRICS:          AGGREGATED numbers over time ("average tool
                 accuracy this week is 91%" -- directly Module 20's
                 aggregation)

TRACING:              the COMPLETE, connected sequence of events for
                     ONE specific agent run -- showing the
                     full trajectory, not just isolated events

These are complementary, not redundant: logging gives you individual data points, metrics give you trends, and tracing gives you the full story of one specific execution — exactly what’s needed to actually run Module 19’s diagnostic process against a real, specific failure.

5. An Example Agent Trace

sequenceDiagram
    participant U as User
    participant A as Agent
    participant T1 as check_order_status
    participant T2 as check_shipping_carrier
    U->>A: "My order is late"
    A->>A: Thought: check order status first
    A->>T1: call(order_id=4471)
    T1-->>A: "Order is LATE"
    A->>A: Thought: check shipping carrier next
    A->>T2: call(tracking=1Z999)
    T2-->>A: "Delivered 2 days ago"
    A->>U: "Escalating to claims"

This trace directly shows the SAME information Module 9’s ReAct pattern produces — observability is precisely the practice of capturing and storing this trace, not just producing it transiently.

6. A Real Developer Example

TechCorp reviews a production incident using their agent trace:

Log EntryWhat It Reveals
reasoning: "checking order status"Confirms the agent’s first decision was reasonable
tool_call: check_order_status → "Order is LATE"Confirms the tool returned correct data
state_update: order_status=lateConfirms state was updated correctly
tool_call: check_shipping_carrier → TIMEOUTReveals the actual failure point — the carrier API timed out
reasoning: "retrying shipping carrier check"Shows the agent’s real retry behavior
tool_call: check_shipping_carrier → TIMEOUT (again)Confirms this is a persistent failure, not a fluke

Without this trace, TechCorp would only know “the agent failed to resolve the ticket” — with it, they know exactly which tool call failed, and can fix the actual root cause (directly Module 19’s diagnostic pattern, now applied to real, captured data).

7. A Simple Agentic AI Connection

Observability directly connects to Module 15’s multi-agent systems — a real multi-agent trace needs to capture not just what happened, but which specific agent was responsible for each step, exactly extending this module’s single-agent trace structure to attribute each event to its originating agent.

8. How Is This Used in AI?

🤖 How Is This Used in AI?

Production agent systems implement observability as a real, foundational requirement — not an optional add-on — precisely because Module 19’s diagnostic process and Module 20’s evaluation practice both fundamentally depend on having real, captured trajectory data to work with, rather than only the final output.

9. Real-World Applications

  • Debugging production incidents by reviewing the exact trace leading to a failure
  • Feeding captured trajectories into Module 20’s evaluation pipeline
  • Monitoring dashboards tracking real-time cost, latency, and error rates across many concurrent agent runs

10. Common Mistakes

Incorrect idea: Only logging errors, not the full reasoning trajectory.

Why it is incorrect: As shown directly in Section 2, this makes Module 19’s diagnostic process impossible to apply to real production failures.

Incorrect idea: Conflating logging, metrics, and tracing as the same thing.

Why it is incorrect: As shown directly in Section 4, each serves a different, complementary purpose.

Incorrect idea: Not capturing WHICH agent was responsible in a multi-agent system.

Why it is incorrect: As shown directly in Section 7, this makes diagnosing multi-agent failures much harder.

11. Limitations

  • Comprehensive tracing adds storage and processing overhead — a real trade-off against the diagnostic value it provides
  • Even a complete trace doesn’t automatically explain WHY a specific reasoning decision was made — it shows WHAT happened, but real interpretation still requires human (or LLM-judge) review

12. Quick Reference

flowchart TD
    Run[One Agent Run] --> Trace[Full Trace:<br/>reasoning + tools + state]
    Trace --> Log[Individual Log Events]
    Log --> Metrics[Aggregated Metrics<br/>over many runs]
    Trace --> Diag[Module 19's Diagnostic Process]
    Metrics --> Eval[Module 20's Evaluation]

13. Code — Implementing an AI Agent Tracer

🎯 Target of this example: implement Section 6’s real developer example directly — capturing decision summaries, tool calls, and state updates as a complete, structured trace, exactly Section 3’s requirements made into working, inspectable code.

Example 1 — Simple

from dataclasses import dataclass
from datetime import datetime

@dataclass
class TraceEvent:
    event_type: str
    detail: dict
    timestamp: str

class AgentTracer:
    """Captures a real agent trace -- every reasoning step, tool
    call, and state transition (Section 3) -- directly addressing
    why ordinary application logging is insufficient (Section 2)."""

    def __init__(self):
        self.events: list = []

    def log(self, event_type: str, detail: dict):
        self.events.append(TraceEvent(event_type, detail, datetime.now().isoformat()))

    def log_reasoning(self, thought: str):
        self.log("reasoning", {"thought": thought})

    def log_tool_call(self, tool: str, arguments: dict, result: str):
        self.log("tool_call", {"tool": tool, "arguments": arguments, "result": result})

    def log_state_update(self, key: str, value: str):
        self.log("state_update", {"key": key, "value": value})

    def summary(self) -> dict:
        return {
            "total_events": len(self.events),
            "reasoning_steps": sum(1 for e in self.events if e.event_type == "reasoning"),
            "tool_calls": sum(1 for e in self.events if e.event_type == "tool_call"),
            "state_updates": sum(1 for e in self.events if e.event_type == "state_update"),
        }

tracer = AgentTracer()
tracer.log_reasoning("I need to check the order status first.")
tracer.log_tool_call("check_order_status", {"order_id": "4471"}, "Order is LATE")
tracer.log_state_update("order_status", "late")
tracer.log_reasoning("The order is late, checking shipping carrier.")
tracer.log_tool_call("check_shipping_carrier", {"tracking": "1Z999"}, "Delivered 2 days ago")

print(tracer.summary())
for event in tracer.events:
    print(f"  [{event.event_type}] {event.detail}")

Expected Output:

{'total_events': 5, 'reasoning_steps': 2, 'tool_calls': 2,
'state_updates': 1}
  [reasoning] {'thought': 'I need to check the order status first.'}
  [tool_call] {'tool': 'check_order_status', 'arguments': {'order_id':
'4471'}, 'result': 'Order is LATE'}
  [state_update] {'key': 'order_status', 'value': 'late'}
  [reasoning] {'thought': 'The order is late, checking shipping
carrier.'}
  [tool_call] {'tool': 'check_shipping_carrier', 'arguments':
{'tracking': '1Z999'}, 'result': 'Delivered 2 days ago'}

What we conclude from this example: the tracer captures EVERY real event type from Section 3’s requirements — reasoning, tool calls, and state updates — with a complete summary count, exactly the structured trace a real production system needs to make Module 19’s diagnostic process actually possible.

Example 2 — Intermediate

from dataclasses import dataclass
from datetime import datetime

@dataclass
class TraceEvent:
    event_type: str
    detail: dict
    timestamp: str

class DiagnosticTracer:
    """Extends Example 1 to directly implement Section 6's REAL
    developer example -- surfacing the EXACT failure point (a
    persistent tool timeout) from a captured trace."""

    def __init__(self):
        self.events: list = []

    def log(self, event_type: str, detail: dict):
        self.events.append(TraceEvent(event_type, detail, datetime.now().isoformat()))

    def find_failures(self) -> list:
        """Directly implements Section 6's diagnostic value --
        scanning the trace for real failure signals."""
        return [e for e in self.events if e.event_type == "tool_call" and "TIMEOUT" in e.detail.get("result", "")]

tracer = DiagnosticTracer()
tracer.log("reasoning", {"thought": "checking order status"})
tracer.log("tool_call", {"tool": "check_order_status", "result": "Order is LATE"})
tracer.log("state_update", {"key": "order_status", "value": "late"})
tracer.log("tool_call", {"tool": "check_shipping_carrier", "result": "TIMEOUT"})
tracer.log("reasoning", {"thought": "retrying shipping carrier check"})
tracer.log("tool_call", {"tool": "check_shipping_carrier", "result": "TIMEOUT"})

failures = tracer.find_failures()
print(f"Total events in trace: {len(tracer.events)}")
print(f"Failure events found: {len(failures)}")
for f in failures:
    print(f"  Tool '{f.detail['tool']}' failed with: {f.detail['result']}")

Expected Output:

Total events in trace: 6
Failure events found: 2
  Tool 'check_shipping_carrier' failed with: TIMEOUT
  Tool 'check_shipping_carrier' failed with: TIMEOUT

What we conclude from this example: scanning the captured trace correctly identifies BOTH timeout occurrences, revealing this is a persistent failure (not a one-time fluke) — exactly Section 6’s real developer example, where the trace reveals the exact, specific root cause a team would otherwise have to guess at.

Example 3 — Production Grade

from dataclasses import dataclass, field
from datetime import datetime

@dataclass
class TraceEvent:
    event_type: str
    detail: dict
    timestamp: str
    agent_id: str = "default"

class MultiAgentTracer:
    """A production-style tracer implementing Section 7's requirement
    -- attributing EVERY event to its ORIGINATING agent, directly
    supporting diagnosis in a multi-agent system (Module 15) where
    knowing WHICH agent caused a problem is essential."""

    def __init__(self):
        self.events: list = []

    def log(self, agent_id: str, event_type: str, detail: dict):
        self.events.append(TraceEvent(event_type, detail, datetime.now().isoformat(), agent_id))

    def events_by_agent(self, agent_id: str) -> list:
        return [e for e in self.events if e.agent_id == agent_id]

    def cost_by_agent(self) -> dict:
        costs = {}
        for e in self.events:
            costs[e.agent_id] = costs.get(e.agent_id, 0.0) + e.detail.get("cost", 0.0)
        return costs

tracer = MultiAgentTracer()
tracer.log("researcher", "tool_call", {"tool": "search_docs", "cost": 0.01})
tracer.log("coder", "tool_call", {"tool": "write_code", "cost": 0.03})
tracer.log("coder", "reasoning", {"thought": "refining implementation", "cost": 0.02})
tracer.log("reviewer", "tool_call", {"tool": "run_tests", "cost": 0.005})

costs = tracer.cost_by_agent()
print("Cost breakdown by agent:")
for agent_id, cost in costs.items():
    print(f"  {agent_id}: ${cost:.3f}")

coder_events = tracer.events_by_agent("coder")
print(f"\nCoder agent's events: {len(coder_events)}")

Expected Output:

Cost breakdown by agent:
  researcher: $0.010
  coder: $0.050
  reviewer: $0.005

Coder agent's events: 2

What we conclude from this example: the tracer correctly attributes cost to each SPECIFIC agent in a multi-agent system — immediately revealing that the coder agent consumed the most resources — exactly Section 7’s requirement, made into working, queryable observability data that a real team could use to identify which specific agent in a coordinated system needs optimization or debugging attention.

14. Interview Questions

Q: Why is only logging errors or final outputs insufficient for agent observability?

Ans: An agent’s operation involves multiple reasoning steps, tool calls, and state transitions, all of which are necessary to understand why a specific final result was produced. Logging only errors or final outputs tells you that something went wrong, but not where in the trajectory the actual problem occurred — you can’t apply a systematic diagnostic process without visibility into the full sequence of decisions and actions the agent actually took.

Q: Distinguish logging, metrics, and tracing as three related but different observability concepts.

Ans: Logging captures individual, discrete events — like a specific tool being called with specific arguments. Metrics are aggregated numbers over time, like average tool accuracy across many runs. Tracing is the complete, connected sequence of events for one specific agent run, showing the full trajectory rather than isolated data points. These are complementary — logs give granular data points, metrics reveal trends, and traces tell the complete story of a specific execution needed for detailed diagnosis.

Q: Using the TechCorp shipping-carrier example, explain how a captured trace enabled a diagnosis that wouldn’t have been possible otherwise.

Ans: The trace showed the agent’s reasoning was sound at each step — checking order status first, correctly updating state — until the check_shipping_carrier tool call returned a timeout. It also showed the agent retried and hit the same timeout again, revealing this was a persistent failure of that specific external API, not a fluke. Without this trace, the team would only know the ticket wasn’t resolved, with no way to determine that the actual root cause was a specific external system’s reliability problem rather than a flaw in the agent’s own reasoning.

Q: Why does a multi-agent system’s observability need to capture which specific agent was responsible for each event?

Ans: In a multi-agent system, a failure or cost overrun could originate from any of several agents working together on a task. Without attributing each traced event to its specific originating agent, diagnosing a problem becomes much harder — you’d know something went wrong in the overall system, but not which specific agent’s reasoning, tool call, or cost consumption actually caused it. Tagging events by agent lets a team isolate the specific component responsible, directly enabling targeted debugging or optimization.

15. What You Should Remember

  • A real agent trace must capture reasoning, tool calls, state transitions, and cost — not just the final answer — verified directly through a tracer capturing all of these event types with a complete, queryable summary.
  • Logging, metrics, and tracing are complementary, distinct concepts — individual events, aggregated trends, and complete per-run trajectories respectively.
  • A captured trace enables real diagnosis that final-output-only logging cannot — verified directly by identifying a persistent tool timeout as the specific root cause of a failure, and by attributing cost to specific agents in a multi-agent system.

16. Quick Practice

Design a trace schema (following Section 3’s element list) for an agent in your own domain, specifying exactly what fields you’d capture for each event type — and identify one real production scenario where having this trace would let you diagnose a problem that final-output-only logging would have missed.

17. Next Step

Next: Module 22 — Agent Frameworks — introducing LangChain and LangGraph only now, after understanding the manual orchestration problems they exist to solve, mapping every framework concept back to something you’ve already built from scratch.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed