TechByteByByte

Observability

Level 5 begins here: AI-specific observability beyond traditional logs and metrics — prompt tracing, token usage, per-stage latency, and debugging one failed AI request end-to-end.

#AI Engineering#Observability#Level 5

Begin with the problem

A failed AI response can come from retrieval, context assembly, the model, a tool, or output handling. Observability records enough evidence to find the first stage that went wrong.

request → trace stages + tokens + cost + latency → diagnose → connect to evaluation

What you will learn

  • Separate logs, metrics, and traces.
  • Capture prompts safely together with retrieval, tools, timing, tokens, and versions.
  • Use trace evidence without leaking sensitive data.

Current production grounding: OpenAI’s Evals documentation shows dataset- and grader-based evaluation for model applications.

These links document current public features and practices. They do not reveal a provider’s private implementation or guarantee that every product uses identical defaults.

Topic-specific reference: OpenTelemetry’s GenAI semantic conventions define attributes for model operations, data sources, evaluation scores, and model messages, while warning that message content may contain sensitive data.

1. The Engineering Problem

You know how to log traditional application requests. But “the API returned a 200” tells you nothing about why an AI system gave a specific answer, how much that specific request cost, or which of its several internal stages (retrieval, model call, tool execution) took the most time.

This module covers what a AI-aware observability layer needs to capture — directly enabling Module 7’s RAG diagnostics and Module 10’s evaluation, both of which depend on this data existing in the first place.

2. Why Traditional Observability Falls Short

TRADITIONAL observability answers:      did the request succeed?
                                       how long did it take? what
                                       was the error?

AI observability also needs to answer:      WHAT was
                                                      retrieved? HOW
                                                      MANY tokens
                                                      were used? WHAT
                                                      did the model
                                                      actually see?
                                                      HOW MUCH did
                                                      this specific
                                                      request cost?

A traditional APM tool doesn’t know what a “token” is or that “retrieval” is a distinct internal stage worth tracking separately from “the model call.” AI observability is a superset of traditional observability, purpose-built for these AI-specific dimensions.

3. The AI Observability Dimensions

DimensionWhat It Captures
Prompt tracingThe exact, final prompt sent to the model (Module 5-6)
Token usagePrompt tokens, completion tokens — directly feeds cost (Module 16)
Time to first token (TTFT)Latency until the model starts streaming a response
Tokens per secondGeneration speed once streaming begins
Retrieval traceWhich documents were retrieved, their scores, which survived into context (Module 6-7)
Tool callsWhich tools were invoked, with what arguments, and what they returned (Module 9)
Cost per requestreal dollar cost, computed per request, not just at the monthly bill
Evaluation scoresIf lightweight real-time evaluation runs, its result attached to this specific request

4. A Real-World Analogy — The Airport, Revisited

Module 3's airport analogy: OBSERVABILITY is the airport's
BLACK BOX RECORDER and LIVE RADAR TRACKING.

A traditional flight log might record "flight departed, flight
arrived, on time." An AI-aware equivalent records EVERY leg of the
journey: altitude, speed, fuel burn, weather consulted, EVERY
instrument reading -- so that if something goes wrong,
investigators can reconstruct EXACTLY what happened, at EVERY stage,
not just "it didn't arrive as expected."

5. Debugging One Failed Request, End-to-End

A user reports: "the assistant gave me a wrong answer about my
return."

WITHOUT tracing: you can only ask the user to repeat the
                        question and hope it fails again -- you
                        have NO record of what ACTUALLY happened.

WITH tracing, you can directly answer:

  - What was the ORIGINAL query?
  - What documents were RETRIEVED, and with what scores?
  - Which documents survived into the FINAL context (Module 6)?
  - What was the EXACT prompt sent to the model?
  - What did the model ACTUALLY generate?
  - Did a real-time evaluation check flag anything?

  -- directly Module 7's RAG diagnostic process, made POSSIBLE only
  because this data was captured in the first place

6. A worked developer example

TechCorp’s trace for one support request:

FieldValue
request_idreq_4471
retrieved_doc_ids["policy_doc_12", "faq_entry_3"]
retrieval_latency_ms45.2
prompt_tokens / completion_tokens850 / 120
model_latency_ms620.5
total_latency_ms665.7
cost_usd0.001515

When a customer later disputes this specific answer, TechCorp’s support team can pull up this exact trace and see precisely what the model was shown — resolving the dispute in minutes instead of being unable to investigate at all.

7. How Is This Used in the Industry?

🤖 How Is This Used in the Industry?

Production AI teams implement distributed tracing specific to their AI orchestration layer (Module 3) — every request gets a trace ID that follows it through retrieval, model calls, and tool executions, feeding dashboards that track cost, latency, and quality trends over time, and enabling exactly the kind of request-level investigation Section 5 describes.

8. Common Mistakes

Incorrect idea: Logging only the final response, not the intermediate stages.

Why it is incorrect: This shortcut removes a validation or control boundary, allowing an error to pass into later stages where it becomes harder and more expensive to detect. As shown directly in Section 5, this makes root-cause debugging impossible after the fact.

Incorrect idea: Not tracking cost per request, only at the monthly bill.

Why it is incorrect: As shown directly in Module 16’s discussion, this makes it impossible to identify which specific requests or features are actually expensive.

Incorrect idea: Treating AI observability as identical to traditional APM.

Why it is incorrect: As shown directly in Section 2, AI systems need additional, specific dimensions traditional tools don’t capture by default.

9. Code — A Minimal AI-Aware Request Tracer

What this shows: a working tracer capturing exactly Section 3’s dimensions — retrieval detail, token usage, per-stage latency, and cost — attributable to one specific request, exactly Section 6’s worked developer example made concrete.

from dataclasses import dataclass, field

@dataclass
class RequestTrace:
    request_id: str
    prompt_tokens: int = 0
    completion_tokens: int = 0
    retrieval_latency_ms: float = 0.0
    model_latency_ms: float = 0.0
    total_latency_ms: float = 0.0
    retrieved_doc_ids: list = field(default_factory=list)
    cost_usd: float = 0.0

class AITracer:
    """An AI-specific tracer -- captures the dimensions
    traditional application observability doesn't (Section 2):
    tokens, retrieval detail, per-stage latency, and cost."""

    PRICE_PER_1K_INPUT = 0.0015
    PRICE_PER_1K_OUTPUT = 0.002

    def __init__(self):
        self.traces: list = []

    def start_trace(self, request_id: str) -> RequestTrace:
        trace = RequestTrace(request_id=request_id)
        self.traces.append(trace)
        return trace

    def record_retrieval(self, trace: RequestTrace, doc_ids: list, latency_ms: float):
        trace.retrieved_doc_ids = doc_ids
        trace.retrieval_latency_ms = latency_ms

    def record_model_call(self, trace: RequestTrace, prompt_tokens: int, completion_tokens: int, latency_ms: float):
        trace.prompt_tokens = prompt_tokens
        trace.completion_tokens = completion_tokens
        trace.model_latency_ms = latency_ms
        # Cost computed per request, NOT only visible at the monthly
        # bill (Section 8's common mistake, directly addressed).
        trace.cost_usd = round(
            (prompt_tokens / 1000) * self.PRICE_PER_1K_INPUT +
            (completion_tokens / 1000) * self.PRICE_PER_1K_OUTPUT, 6
        )

    def finalize(self, trace: RequestTrace):
        trace.total_latency_ms = trace.retrieval_latency_ms + trace.model_latency_ms

tracer = AITracer()
trace = tracer.start_trace("req_4471")
tracer.record_retrieval(trace, doc_ids=["policy_doc_12", "faq_entry_3"], latency_ms=45.2)
tracer.record_model_call(trace, prompt_tokens=850, completion_tokens=120, latency_ms=620.5)
tracer.finalize(trace)

print(f"Request: {trace.request_id}")
print(f"Retrieved docs: {trace.retrieved_doc_ids}")
print(f"Total latency: {trace.total_latency_ms}ms")
print(f"Cost: ${trace.cost_usd}")

Expected Output:

Request: req_4471
Retrieved docs: ['policy_doc_12', 'faq_entry_3']
Total latency: 665.7ms
Cost: $0.001515

What this confirms: exactly Section 6’s TechCorp trace values are reproduced, attributable to one specific request ID — the kind of complete record that turns “the assistant gave a wrong answer” from an undiagnosable complaint into a request that can be pulled up and investigated directly, exactly Section 5’s debugging scenario made possible.

10. Production Considerations

  • Trace data needs a retention policy — storing every field for every request forever is expensive; most teams retain full detail for a shorter window and aggregate metrics longer term
  • Ensure trace data doesn’t itself become a privacy risk — logging full prompts/responses containing PII needs the same handling discipline as Module 13 (Security) covers for the data itself

11. Trade-offs

  • Capturing this level of detail adds real storage and processing overhead per request — worthwhile given the debugging and cost-accounting value it provides
  • Real-time evaluation checks attached to every trace (Section 3) add latency — many teams run this on a sample of traffic rather than every single request

12. Chapter Summary

AI observability is a superset of traditional application observability, purpose-built to capture what’s specific to AI systems: prompt content, token usage, retrieval detail, tool calls, and per-request cost. Without this data captured at request time, a reported bad answer is undiagnosable after the fact — you can only guess.

With it, you can reconstruct exactly what the model saw and did for any specific request, directly enabling the diagnostic and cost-accounting work covered throughout this course.

13. Visual Cheat Sheet

Traditional observability: success/fail, latency, error

AI observability ADDS:  prompt content | token usage | TTFT/TPS |
                        retrieval trace | tool calls | cost/request
                        | eval scores

14. Top Takeaways

  1. AI observability is a superset of traditional observability — it needs dimensions (tokens, retrieval, cost) that traditional APM tools don’t capture by default.
  2. Debugging a specific bad AI response requires the full intermediate trace — retrieval, exact prompt, model output — not just the final response.
  3. Cost should be tracked per request, not only visible at the monthly bill.
  4. Trace data needs a retention policy and the same privacy handling discipline as the underlying data itself.
  5. This trace data directly enables Module 7’s RAG diagnostics and Module 10’s evaluation — observability is foundational, not optional.

15. Interview Questions

Q: 1. Why is traditional application observability (logs, metrics, traces) insufficient for an AI system?**

Ans: Traditional observability answers whether a request succeeded and how long it took, but doesn’t capture AI-specific dimensions necessary for debugging and cost accounting — what was actually retrieved, the exact prompt sent to the model, token usage, and per-request cost. Without these, a reported bad response is undiagnosable after the fact.

  • Why it matters: Teams relying only on traditional observability can tell THAT something went wrong but can’t determine WHY.
  • Real-world example: Section 5’s debugging scenario — reconstructing exactly what a model saw for a disputed answer.
  • Common mistake: Assuming existing APM tooling is sufficient without adding AI-specific tracing.
  • Interviewer is testing: Whether the candidate understands the specific gap between traditional and AI observability.
  • Likely follow-up: “What would you do if trace data revealed the retrieval step returned the wrong documents?” → This becomes Module 7’s RAG diagnostic process, now actionable because the data exists.

Q: 2. Why should cost be tracked per request rather than only observed in the monthly bill?**

Ans: A monthly aggregate tells you total spend but nothing about WHICH requests, features, or users are driving that cost. Per-request cost tracking lets you identify expensive patterns — an inefficient prompt, an oversized model used unnecessarily (Module 4), or a specific feature with runaway usage — and address them directly, rather than discovering a cost problem a month after it started.

  • Why it matters: This directly enables the cost engineering practices covered in Module 16 — you can’t optimize what you can’t measure at the right granularity.
  • Real-world example: Section 6’s TechCorp trace, showing cost attributed to one specific, identifiable request.
  • Common mistake: Only discovering a cost spike when the monthly bill arrives, with no way to attribute it to a specific cause.
  • Interviewer is testing: Whether the candidate connects observability directly to cost control, not just debugging.
  • Likely follow-up: “How would you use this data to catch a cost spike early?” → Alerting on aggregate cost trends computed from per-request data, catching anomalies before the monthly bill.

16. Scenario-Based Question

Scenario: A customer disputes an answer TechCorp’s assistant gave about a refund amount, claiming it was wrong. The support team checks their logs and finds only “request succeeded, 200 OK, 620ms” — no record of what the assistant actually retrieved or generated.

  • Problem Analysis: Section 8’s common mistake — only the final success/fail status was logged, none of the AI-specific intermediate detail.
  • How to Think: This isn’t a “the assistant was wrong” problem to investigate yet — it’s an observability gap preventing any investigation at all.
  • Investigation: Without a trace, there’s nothing to investigate for THIS specific past request — the team can only ask the customer to reproduce the issue and hope it recurs.
  • Root Cause: No AI-specific request tracing (Section 3) was implemented — only traditional success/fail logging.
  • Solution: Implement Section 9’s tracer immediately, capturing retrieval detail, exact prompt, and generated output for every future request; for THIS specific dispute, the team must rely on the customer’s account alone.
  • Trade-offs: Adding full tracing retroactively means this specific incident can’t be resolved with certainty — a real cost of not having built observability in from the start.
  • Production Considerations: This scenario is exactly why Section 12 treats observability as foundational rather than optional — the cost of NOT having it becomes concretely visible only once a real dispute like this one occurs, by which point it’s too late for that specific case.

17. Next Step

Next: Module 13 — AI Security — a comprehensive treatment of prompt injection, data leakage, tool abuse, and security as a architecture, not just a list of attacks to defend against individually.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed