TechByteByByte

Context Engineering

Closing Level 2: why context is one of the most important, scarce resources in an AI system, and how to select, compress, order, and prioritize it deliberately — distinct from prompt engineering.

#AI Engineering#Context Engineering#Level 2

Begin with the problem

A model can only use the information placed in its context window. Context engineering decides what information earns that limited space, in what order, and with what trust level.

available information → select → compress/order → context window → model response

What you will learn

  • Distinguish prompt instructions from contextual information.
  • Select, order, compress, and label context deliberately.
  • Balance relevance, token budget, freshness, and security.

Current production grounding: OpenAI’s Evals documentation shows dataset- and grader-based evaluation for model applications.

Current production grounding: Google’s Gemini tools documentation distinguishes managed built-in tools from custom functions executed by the application.

These links document current public features and practices. They do not reveal a provider’s private implementation or guarantee that every product uses identical defaults.

1. The Engineering Problem

Module 5 treated the prompt as a structured artifact. This module zooms in on one specific, scarce input to that prompt: context — the retrieved documents, conversation history, tool results, and memory a model actually reasons over.

A context window is finite, and every token you put in it is a deliberate, resource-allocation decision, not a free container to dump everything into.

2. Prompt Engineering vs. Context Engineering

PROMPT ENGINEERING:      HOW you instruct the model -- wording,
                        structure, examples (Module 5)

CONTEXT ENGINEERING:         WHAT information the model actually
                            sees when reasoning -- retrieved
                            documents, history, tool results, memory

These are distinct, complementary disciplines. A perfectly worded prompt with the WRONG or too much context still produces a bad answer — and a well-selected, well-ordered context with a mediocre prompt often still produces a good one. Context is frequently the higher-leverage lever.

3. Why Context Is a Scarce Resource

A context window is FINITE -- and every additional token:

  - Costs real money (Module 16)
  - Adds real latency (Module 17)
  - Risks "lost in the middle" -- relevant content buried
    among less relevant content, reducing the model's effective use
    of it

The naive instinct — “just include everything, the context window is big” — is counterproductive. More context is not automatically better context; it’s more tokens competing for the model’s attention, at real cost.

4. The Context Engineering Operations

OperationWhat It Does
SelectionChoosing WHICH pieces of available information belong in this specific request’s context
CompressionReducing verbose content to its essential information (summarizing a long document instead of including it whole)
OrderingPlacing the MOST relevant content where the model uses it most reliably (typically first or last, not buried in the middle)
PrioritizationWhen context exceeds budget, deciding what to drop first

5. A Real-World Analogy — The Hospital

A DOCTOR reviewing a patient's chart before a consultation doesn't
read EVERY note ever recorded about every patient in the hospital --
a well-run system SELECTS the relevant recent history,
COMPRESSES old notes into summaries, and ORDERS the most critical
information (allergies, current medications) FIRST, where it's
most likely to be seen and acted on.

Handing the doctor the ENTIRE hospital's records "just in case"
would be worse, not better -- exactly Section 3's point.

6. What Goes Into Context

CONVERSATION HISTORY:      recent, relevant turns -- not
                          necessarily the ENTIRE conversation
                          (Module 19, application memory)

RETRIEVED CONTEXT:             from RAG (Module 7) -- the
                              most relevant chunks, not everything
                              that matched at all

TOOL RESULTS:                      the OUTPUT of actions
                                  the system already took this turn

MEMORY:                                relevant, PERSISTED
                                      facts from prior sessions
                                      (Module 19)

Each of these competes for the same limited token budget — context engineering is precisely the discipline of allocating that budget deliberately across all four.

7. Context Pollution — A Real Failure Mode

CONTEXT POLLUTION: irrelevant, stale, or redundant content
                   crowding out what actually matters.

Example: including the FULL text of a 50-page policy manual when
        only ONE paragraph is relevant to the user's
        question -- the model now has to find the signal in far
        more noise than necessary.

Important clarification: This directly connects to your RAG course’s “more retrieved chunks isn’t automatically better” principle — context engineering is that same discipline, applied to EVERY source of context, not just retrieval.

Why it matters: This shortcut removes a validation or control boundary, allowing an error to pass into later stages where it becomes harder and more expensive to detect.

8. A worked developer example

TechCorp’s support assistant assembling context for one request:

Available ContentRelevanceIncluded?Why
Retrieved return policy chunkHigh✅ YesDirectly answers the question
Retrieved FAQ entry about refund timingModerate✅ Yesrelated, adds useful detail
Retrieved shipping policy chunkLow❌ No (or last, if budget allows)Not relevant to THIS question
Full marketing copy about a new productNear zero❌ Nopollutes context, no value here

This is context selection and prioritization, applied concretely to one real request.

9. How Is This Used in the Industry?

🤖 How Is This Used in the Industry?

Production RAG and agent systems implement explicit context-assembly logic in the orchestration layer (Module 3) — scoring, filtering, and ordering every candidate piece of context before it reaches the prompt, rather than concatenating everything available and hoping the model sorts it out.

10. Common Mistakes

Incorrect idea: Including entire documents when only a fraction is relevant.

Why it is incorrect: As shown directly in Section 7, this is context pollution, not thoroughness.

Incorrect idea: Ignoring “lost in the middle” effects when ordering context.

Why it is incorrect: This shortcut removes a validation or control boundary, allowing an error to pass into later stages where it becomes harder and more expensive to detect. As shown directly in Section 3, placement affects how reliably the model uses information.

Incorrect idea: Treating conversation history as something to always include in full.

Why it is incorrect: As shown directly in Section 6, recent and relevant turns matter more than the complete, unfiltered history.

11. Code — A Context Selection and Ordering Function

What this shows: implementing Section 4’s selection and ordering operations directly — a function that picks the most relevant available context within a token budget, ordered so the highest-relevance content comes first, exactly Section 8’s real developer example made concrete.

from dataclasses import dataclass

@dataclass
class ContextItem:
    source: str
    content: str
    relevance_score: float
    token_estimate: int

def select_and_order_context(items: list, token_budget: int) -> tuple:
    """Directly implements CONTEXT SELECTION and ORDERING (Section
    4) -- picks the most relevant items that fit the budget, ordered
    with the MOST relevant first (avoiding 'lost in the middle')."""
    sorted_items = sorted(items, key=lambda i: i.relevance_score, reverse=True)

    selected = []
    used_tokens = 0
    for item in sorted_items:
        if used_tokens + item.token_estimate <= token_budget:
            selected.append(item)
            used_tokens += item.token_estimate

    return selected, used_tokens

# Exactly Section 8's TechCorp example, as real candidate context
items = [
    ContextItem("policy_doc_1", "Return policy: 30 days.", relevance_score=0.92, token_estimate=50),
    ContextItem("policy_doc_2", "Shipping policy: 5-7 business days.", relevance_score=0.35, token_estimate=60),
    ContextItem("faq_entry_3", "Refunds processed within 3-5 days.", relevance_score=0.78, token_estimate=40),
    ContextItem("marketing_copy", "Check out our new product line!", relevance_score=0.05, token_estimate=200),
]

selected, used = select_and_order_context(items, token_budget=150)
print(f"Selected {len(selected)} of {len(items)} items, using {used}/150 tokens:")
for item in selected:
    print(f"  [{item.relevance_score}] {item.source}: {item.content}")

Expected Output:

Selected 3 of 4 items, using 150/150 tokens:
  [0.92] policy_doc_1: Return policy: 30 days.
  [0.78] faq_entry_3: Refunds processed within 3-5 days.
  [0.35] policy_doc_2: Shipping policy: 5-7 business days.

What this confirms: the irrelevant marketing copy (relevance 0.05) is correctly excluded despite fitting within budget alone, while the three relevant items are selected and ordered highest-relevance-first — exactly Section 4’s operations, made into real, working selection logic rather than an ad-hoc “include everything that fits” approach.

12. Production Considerations

  • Token estimates should use the actual tokenizer for your model, not a rough word count — Module 16 covers precise token accounting
  • Context assembly logic belongs in the orchestration layer (Module 3), centralized rather than duplicated per feature

13. Trade-offs

  • Aggressive compression saves tokens and cost but risks losing nuance the model might have used — the right compression level is a task-specific judgment call
  • More sophisticated relevance scoring (e.g., reranking, your RAG course’s Module 18) adds real latency in exchange for better selection quality

14. Chapter Summary

Context engineering is distinct from prompt engineering — it’s the discipline of deliberately selecting, compressing, ordering, and prioritizing what information a model actually sees, given a finite, real token budget. More context is not automatically better; irrelevant or poorly-ordered context pollutes the model’s effective attention and wastes real cost and latency.

Every source of context — retrieval, history, tool results, memory — competes for the same limited budget and needs to be allocated deliberately.

15. Visual Cheat Sheet

Available context (retrieval + history + tools + memory)
        |
   SELECT (relevant only)
        |
   COMPRESS (essential information)
        |
   ORDER (most relevant FIRST, avoid "lost in the middle")
        |
   PRIORITIZE (drop lowest-value content first if over budget)
        |
   Final context -> prompt (Module 5)

16. Top Takeaways

  1. Context engineering (what the model sees) is distinct from prompt engineering (how you instruct it).
  2. A context window is a finite, costly resource — not a free container.
  3. More context is not automatically better — irrelevant content is context pollution, not thoroughness.
  4. Ordering matters — place the most relevant content where the model uses it most reliably, not buried in the middle.
  5. Every context source (retrieval, history, tools, memory) competes for the same limited budget and needs deliberate allocation.

17. Interview Questions

Q: 1. Distinguish prompt engineering from context engineering.**

Ans: Prompt engineering concerns how you instruct the model — wording, structure, examples. Context engineering concerns what information the model actually sees — the selected, compressed, and ordered retrieval results, history, tool outputs, and memory.

Both matter, and context is often the higher-leverage lever, since a great prompt with wrong or excessive context still produces a poor answer.

  • Why it matters: Conflating these leads teams to endlessly tweak prompt wording when the problem is what content is being fed to the model.
  • Real-world example: A support assistant giving a hallucinated answer is more often a context problem (wrong or missing retrieved documents) than a prompt-wording problem.
  • Common mistake: Assuming “just adjust the prompt” fixes every quality issue.
  • Interviewer is testing: Whether the candidate can diagnose quality issues at the right layer.
  • Likely follow-up: “How would you determine whether a bad response is a prompt problem or a context problem?” → Module 22’s systematic optimization hierarchy: check context/retrieval before tweaking the prompt further.

Q: 2. Why is “just include more context, the window is big enough” bad advice?**

Ans: More tokens cost more money and add latency, and irrelevant content can dilute the model’s effective attention on what actually matters (context pollution and “lost in the middle” effects). A large context window increases capacity, but doesn’t change the fact that more, less-relevant content is worse than less, more-relevant content.

  • Why it matters: This misconception leads to both higher costs and, counterintuitively, lower quality answers.
  • Real-world example: Including an entire 50-page manual when one paragraph answers the question buries the relevant paragraph among 49 pages of noise.
  • Common mistake: Treating context window size as a target to fill rather than a budget to spend deliberately.
  • Interviewer is testing: Whether the candidate understands context as a scarce resource, not free capacity.
  • Likely follow-up: “How would you decide what to compress versus drop entirely when over budget?” → Prioritize by relevance score (Section 11’s code), compress moderately-relevant content, drop low-relevance content first.

18. Scenario-Based Question

Scenario: TechCorp’s assistant sometimes ignores a critical piece of retrieved information — a specific exception to the standard return policy — even though it’s present in the context sent to the model. Investigation shows this exception is buried as the 7th of 8 retrieved chunks, most of which are only marginally relevant.

  • Problem Analysis: A “lost in the middle” and context pollution issue — Section 3 and 7’s exact failure mode.
  • How to Think: The information WAS present, but its position and the surrounding noise reduced the model’s, effective use of it.
  • Investigation: Review the retrieval and context-assembly logic — is relevance scoring accurate? Is ordering applied at all?
  • Root Cause: No selection/ordering logic — all retrieved chunks are concatenated in retrieval order, not relevance order, with no filtering of low-relevance chunks.
  • Solution: Apply Section 11’s selection-and-ordering approach — filter to relevant chunks only, and order the most relevant (including the critical exception) first.
  • Trade-offs: Requires reliable relevance scoring (potentially a reranking step, adding some latency) — worth it given the alternative is critical information being effectively ignored.
  • Production Considerations: This is exactly why context assembly deserves the same deliberate engineering attention as retrieval itself — retrieval finding the right information isn’t sufficient if context assembly then buries it.

19. Next Step

Next: Module 7 — Production RAG Engineering — Level 3 begins here: building on your RAG course’s foundations to cover the production- specific concerns — caching, observability, and failure analysis — that a working RAG demo doesn’t need but a production RAG system does.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed