TechByteByByte

AI Cost Engineering

Level 6 begins here: what makes AI applications expensive, and the concrete techniques — token budgets, semantic caching, model routing, batching — that control it before the monthly bill becomes a surprise.

#AI Engineering#Cost Engineering#Level 6

Begin with the problem

Token costs that look tiny per request can become a large monthly bill at production volume. Cost engineering measures each stage, then removes waste without silently reducing quality.

tokens/tools/retrieval × traffic → per-request cost → budget → optimize → re-evaluate quality

What you will learn

  • Calculate cost per request and at expected traffic volume.
  • Use routing, caching, batching, and context reduction responsibly.
  • Track cost together with quality rather than optimizing either alone.

Current production grounding: Kubernetes documents workload autoscaling and controlled Deployments for operating containerized services.

These links document current public features and practices. They do not reveal a provider’s private implementation or guarantee that every product uses identical defaults.

1. The Engineering Problem

Traditional software’s marginal cost per request is close to zero — an extra API call costs fractions of a cent in compute. Every AI request, by contrast, costs real, measurable money per token, and that cost compounds directly with traffic, context size, and retry/agent-loop behavior.

Cost isn’t a finance-team afterthought here — it’s a engineering constraint you design against from the start.

2. What Makes AI Applications Expensive

Cost DriverWhy It Adds Up
Input tokensEvery token of context (Module 6) — system prompt, retrieved documents, history — is billed
Output tokenstypically priced higher per token than input
Repeated retrieval/model callsNo caching means paying full price for identical or near-identical requests, every time
Agent loopsEach additional iteration (your Agents course) is another full model call
Embedding/reranking costsreal, recurring costs for every document indexed and every query processed
Oversized model usageModule 4 — using an expensive model for a simple task

3. Token Budgets — A Deliberate Constraint

A TOKEN BUDGET is an explicit limit on how many tokens a
given request type is ALLOWED to consume -- directly connecting to
Module 6's context engineering: context isn't just a quality
concern, it's a DIRECT cost lever.

Treating token usage as an unbounded, “whatever it takes” resource is precisely how costs spiral — a deliberate budget per request type forces, disciplined context and prompt design.

4. Caching — The Single Highest-Leverage Cost Lever

PROMPT CACHING:      many providers let you cache STABLE
                    prompt content (system prompt, few-shot
                    examples) so repeated requests don't pay full
                    processing cost for UNCHANGING content
                    (Module 5, Section 8)

SEMANTIC CACHING:        recognizes when a NEW request is
                        semantically equivalent to a PAST one --
                        even if worded differently -- and reuses the
                        cached response instead of calling the model
                        again

For a knowledge base with repetitive query patterns (common FAQ-style questions), semantic caching alone can eliminate a real, substantial fraction of total model calls — often the single highest-leverage cost optimization available.

5. Model Routing — Module 4, Reframed as Cost Engineering

Module 4's model-selection framework: routing SIMPLE tasks
to a small, cheap model and RESERVING expensive models for complex tasks is VIEWED THROUGH a cost lens, often the single
biggest cost lever available -- because most real production
traffic skews toward simple tasks.

6. Batching and Response Limits

BATCHING:      grouping multiple requests together where
              a provider offers batch pricing -- often significantly
              CHEAPER than real-time, per-request calls, for
              non-time-sensitive workloads

RESPONSE                LIMITS: an explicit cap on output
LENGTH LIMITS:                 length -- preventing a model from
                              generating far more tokens
                              than a task actually needs

7. A Real-World Analogy — The Factory, Revisited

Module 5's factory analogy: a well-run FACTORY doesn't let
raw material USAGE go unmeasured and unbudgeted per product line --
it tracks cost per unit produced, and looks for waste
(REDUNDANT steps, OVERSIZED equipment for a simple task) the same
way COST ENGINEERING looks for redundant model calls and oversized
models.

8. Building a Cost Estimation Model

1. Identify cost drivers for your specific system
   (Section 2)
2. Instrument PER-REQUEST cost tracking (Module 12) -- this is
   the DATA cost engineering runs on
3. Set token budgets per request type (Section 3)
4. Apply CACHING (Section 4) where repetition exists
5. Apply MODEL ROUTING (Section 5) based on task complexity
6. Set ALERTS on aggregate cost trends, not just the monthly bill

9. A worked developer example

TechCorp’s cost optimization, applied to their support assistant:

OptimizationImpact
Semantic caching for FAQ-style questions~40% of total requests served from cache, near-zero marginal cost
Model routing (Module 4) for simple classification tasks~60% of remaining requests use a cheaper model
Token budget on retrieved context (Module 6)prevents context bloat from inflating input-token cost
Per-request cost tracking (Module 12) with alertingCost anomalies caught within hours, not at the monthly bill

10. How Is This Used in the Industry?

🤖 How Is This Used in the Industry?

Mature AI engineering teams treat cost as a first-class, continuously-monitored metric — dashboards track cost per request, per feature, and per user segment, with alerting on anomalies, exactly the same operational discipline applied to infrastructure cost in traditional software.

11. Common Mistakes

Incorrect idea: No caching at all, despite repetitive query patterns.

Why it is incorrect: As shown directly in Section 4, this is often the single highest-leverage optimization left on the table.

Incorrect idea: Using one expensive model for all traffic regardless of task complexity.

Why it is incorrect: As shown directly in Section 5, this is directly Module 4’s mistake, restated as a cost problem.

Incorrect idea: No per-request cost tracking, only discovering cost problems at the monthly bill.

Why it is incorrect: As shown directly in Module 12, this makes root-cause attribution impossible.

12. Code — A Cost Estimator With Semantic Caching

What this shows: a cost-estimation function paired with a minimal semantic cache — directly implementing Section 4’s caching lever and Section 8’s cost-model-building process, exactly Section 9’s worked developer example made concrete.

from dataclasses import dataclass
import hashlib

@dataclass
class CostBreakdown:
    input_tokens: int
    output_tokens: int
    input_cost: float
    output_cost: float
    total_cost: float

def estimate_request_cost(input_tokens: int, output_tokens: int,
                           price_per_1k_input: float, price_per_1k_output: float) -> CostBreakdown:
    """A cost estimation function -- directly implementing
    Module 12's per-request cost tracking (Section 8, step 2)."""
    input_cost = round((input_tokens / 1000) * price_per_1k_input, 6)
    output_cost = round((output_tokens / 1000) * price_per_1k_output, 6)
    return CostBreakdown(input_tokens, output_tokens, input_cost, output_cost, round(input_cost + output_cost, 6))

class SemanticCache:
    """A semantic cache (Section 4) -- normalizes a query so
    that SEMANTICALLY equivalent (not just exact-string-identical)
    requests can reuse a cached response, avoiding a redundant,
    costly model call entirely."""

    def __init__(self):
        self.cache: dict = {}

    def _normalize(self, query: str) -> str:
        # A SIMPLIFIED normalization -- a real system would use
        # embedding SIMILARITY, not just whitespace/case normalization,
        # to catch different phrasings of the same question.
        return " ".join(query.lower().strip().split())

    def get(self, query: str):
        key = hashlib.sha256(self._normalize(query).encode()).hexdigest()
        return self.cache.get(key)

    def set(self, query: str, response: str, cost: float):
        key = hashlib.sha256(self._normalize(query).encode()).hexdigest()
        self.cache[key] = {"response": response, "original_cost": cost}

cache = SemanticCache()

# Estimate cost for a uncached request
cost = estimate_request_cost(input_tokens=850, output_tokens=120, price_per_1k_input=0.0015, price_per_1k_output=0.002)
print(f"Uncached request cost: ${cost.total_cost}")

# Cache the response, then simulate a DIFFERENTLY-formatted but
# semantically identical repeat query
cache.set("What is your return policy?", "Returns accepted within 30 days.", cost.total_cost)
cached_hit = cache.get("  what is your   return policy?  ")

if cached_hit:
    print(f"Cache HIT for reformatted query -- saved ${cached_hit['original_cost']}")
else:
    print("Cache MISS")

Expected Output:

Uncached request cost: $0.001515
Cache HIT for reformatted query -- saved $0.001515

What this confirms: the cache correctly recognizes a differently capitalized, differently spaced query as the SAME underlying request — avoiding a redundant, costly model call entirely — exactly Section 4’s semantic caching lever, made into working code that demonstrates real savings, not just an abstract claim.

13. Production Considerations

  • A real semantic cache needs embedding-based similarity matching (not just normalization, as in Section 12’s simplified example) to catch different phrasings, not just whitespace/case variations
  • Cache invalidation matters — if underlying documents change (Module 7), cached responses referencing outdated information need to expire

14. Trade-offs

  • Caching risks serving a stale response if the underlying knowledge changes — needs invalidation tied to content updates
  • Model routing (Section 5) adds engineering complexity in exchange for real cost savings — worthwhile once traffic volume makes the savings meaningful

15. Chapter Summary

AI cost is a direct function of tokens, model choice, and repeated calls — unlike traditional software’s near-zero marginal request cost. The highest-leverage cost engineering levers are caching (especially semantic caching, avoiding redundant model calls entirely) and model routing (using the cheapest model that meets each task’s quality bar, directly Module 4).

Both require per-request cost tracking (Module 12) as the underlying data these optimizations are measured against — you can’t optimize what you haven’t instrumented.

16. Visual Cheat Sheet

Cost drivers: input tokens + output tokens + repeated calls +
             agent loops + embedding/reranking + oversized models

Highest-leverage levers: SEMANTIC CACHING + MODEL ROUTING
                         (Module 4) + TOKEN BUDGETS (Module 6)

17. Top Takeaways

  1. AI cost is a direct function of tokens and model choice — unlike traditional software’s near-zero marginal request cost.
  2. Semantic caching is often the single highest-leverage cost optimization, since it avoids redundant model calls entirely.
  3. Model routing (Module 4) applied through a cost lens: most real traffic is simple and doesn’t need an expensive model.
  4. Token budgets make context size a deliberate engineering constraint, not an afterthought.
  5. Per-request cost tracking (Module 12) is the data foundation every cost optimization needs to be measured against.

18. Interview Questions

Q: 1. Why is caching often described as the single highest-leverage AI cost optimization?**

Ans: A cache hit avoids a model call entirely — no tokens processed, no cost incurred for that request. For any system with repetitive query patterns (common FAQ-style questions), this can eliminate a real, substantial fraction of total model calls, which is a larger cost reduction than most incremental optimizations on the calls that still happen.

  • Why it matters: Teams often optimize prompt wording or context size before checking whether caching could eliminate a large chunk of calls entirely.
  • Real-world example: Section 9’s TechCorp example — ~40% of requests served from cache.
  • Common mistake: Building semantic caching only after noticing a cost problem, rather than as a default consideration from the start for systems with repetitive traffic.
  • Interviewer is testing: Whether the candidate can identify high-leverage optimizations, not just any valid optimization.
  • Likely follow-up: “What’s the risk with caching, and how would you mitigate it?” → Stale responses if underlying data changes (Section 13) — mitigate with cache invalidation tied to content updates.

Q: 2. Explain how model routing, covered in Module 4 for quality reasons, is also a cost optimization.**

Ans: Model routing sends different tasks to the cheapest model that meets each task’s quality bar.

Since most real production traffic is simple (classification, extraction, short lookups), routing this traffic to small, cheap models while reserving expensive models for complex tasks directly reduces total cost, often substantially, without sacrificing quality on any individual task.

  • Why it matters: Using one expensive model uniformly for all traffic is a common, and easily-avoidable cost mistake.
  • Real-world example: Section 9’s TechCorp example — ~60% of remaining (non-cached) requests routed to a cheaper model.
  • Common mistake: Treating model choice as a single, application- wide setting rather than a per-task, cost-aware decision.
  • Interviewer is testing: Whether the candidate connects quality and cost engineering as related, not separate, concerns.
  • Likely follow-up: “How would you validate that a cheaper model is good enough for a given task?” → Module 10-11’s evaluation framework, run per model candidate before routing traffic to it.

19. Scenario-Based Question

Scenario: TechCorp’s finance team reports the AI system’s monthly cost has grown 3x over six months, while request volume only grew 1.5x. No per-request cost tracking exists — only the aggregate monthly bill.

  • Problem Analysis: Section 11’s common mistake — no per-request cost instrumentation, so the cause of the disproportionate growth can’t be identified from available data.
  • How to Think: Cost growing faster than traffic suggests either increasing per-request cost (larger context, bigger model, more agent iterations) or decreasing cache effectiveness — but without instrumentation, this is a guess, not a diagnosis.
  • Investigation: Immediately implement Module 12’s per-request cost tracking; once data accumulates, break down cost by request type, model used, and cache hit rate over time.
  • Root Cause: Likely candidates include model routing degrading (more traffic routed to an expensive model than originally intended), context size growing (Module 6, unchecked token budgets), or a cache effectiveness regression — can’t be confirmed without the tracking data.
  • Solution: Once tracking reveals the actual driver, apply the specific fix — tighten token budgets (Section 3), restore/improve model routing (Section 5), or fix a cache invalidation bug reducing hit rate (Section 13).
  • Trade-offs: Diagnosis takes real time to gather enough tracked data — a real, unavoidable cost of not having instrumented this from the start.
  • Production Considerations: This scenario is exactly why Section 8’s cost-model-building process places per-request tracking as step 2, before any optimization — you cannot fix what you haven’t measured.

20. Next Step

Next: Module 16 — AI Latency Engineering — where latency comes from across the full request pipeline, and the techniques (streaming, parallel execution, caching) that keep response times within budget.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed