Start with the real problem
Context is the information currently given to the model. Context management decides what to keep, shorten, retrieve, or leave out.
A larger context window is still limited attention and paid input. The challenge is not fitting everything—it is supplying the smallest set of information that changes the answer.
collect context → rank relevance → trim or summarize → assemble → generateWhat you will learn
- Define a context window.
- Separate relevant context from conversation clutter.
- Choose trimming, summarization, retrieval, or state storage.
- Track context cost and information loss.
How this connects to current AI systems
Modern long-context GPT, Gemini, and Claude models still benefit from context selection, caching, and retrieval rather than adding information without checking whether it helps.
1. Why This Module Exists
Module 15 ended with a warning: the full message history is resent every turn, and long conversations eventually hit real limits. This module covers what to actually do about that — and an equally important point: more context isn’t automatically better, even when it fits.
2. The Idea, in Plain Language
Context management means deliberately deciding what information actually goes into the prompt — not just cramming in everything available.
Everything POTENTIALLY relevant:
- Full conversation history
- Every retrieved document
- Every past user preference ever recorded
- Full system instructions
What actually goes in the prompt:
- Only what's RELEVANT to answering THIS specific requestThis is a really important shift in thinking: the goal isn’t “include as much as possible” — it’s “include exactly what’s needed.”
3. Why a Conversation Eventually Becomes a Problem
Every LLM has a context window — a maximum amount of text it can process at once (covered more technically in your LLM course, if you’ve taken it). As a conversation grows turn by turn, the total content being resent grows too, and eventually:
Turn 1: 500 tokens total
Turn 5: 2,000 tokens total
Turn 20: 9,000 tokens total
Turn 50: Exceeds the context limit -- request fails, or something
has to be cutSomething has to give — and how it gets cut matters a great deal.
4. Why More Context Isn’t Automatically Better
Even when everything technically fits, dumping in every possible piece of information isn’t necessarily helpful. Irrelevant context can:
- Distract the AI with information that isn’t actually useful for the current question
- Increase cost proportionally, for no real benefit (Module 25)
- Occasionally confuse the response, especially if some of the included context is outdated or contradicts more recent information
Example — Too much irrelevant context
[Includes the entire 6-month conversation history, including a long
tangent about the user's vacation plans from 3 weeks ago]
User: "What was that marinade recipe you gave me?"Somewhere buried in a huge amount of irrelevant history is the actual answer — but all that noise makes it really harder, not easier, for the AI to reliably locate and use the one relevant detail.
💡 The pattern to notice: the goal is relevant context, not maximum context. This is one of the most counterintuitive but important ideas in this entire module.
5. The Real Strategies for Managing Context
Strategy 1: TRUNCATE Drop the oldest turns, keep only recent
ones. Simple, but can lose relevant early
information.
Strategy 2: SUMMARIZE Periodically compress older turns into a
shorter summary, preserving key facts
while freeing up space.
Strategy 3: RETRIEVE Instead of keeping full history,
store it externally and pull back only
what's specifically relevant to the
CURRENT question. This is the core idea
behind RAG, covered fully in Module 17.Each strategy fits different situations — there’s no single universally “correct” choice.
6. A Real Example From a Developer’s Perspective
A customer support chatbot with long-running conversations needs a real, deliberate context strategy — not just “include everything”:
Naive approach: Always send the FULL conversation history, every
turn, forever.
→ Eventually hits context limits; also gets
expensive and slow as conversations grow.
Better approach: Keep the last 10 turns in full. Periodically
summarize anything older into 2-3 sentences
("Customer previously reported a late order,
which was resolved with a partial refund").
Always include the summary + recent turns, not
the full unbounded history.This keeps the AI aware of important earlier context (via the summary) without the cost and risk of resending an ever-growing, increasingly irrelevant conversation log.
7. A Simple Agentic AI Example
Agents that use tools repeatedly need a real strategy for managing tool results too — not just conversation turns:
"When including past tool results in context, only include the
MOST RECENT result for each tool type, not every single call made so
far in this session. If the user asks about something from much
earlier, look it up again rather than relying on old, potentially
outdated results still sitting in context."This is a really practical agent design decision — old tool results (like a previous “check order status” call) can become stale or irrelevant, and keeping every single one in context forever wastes space and risks the agent acting on outdated information.
8. How Is This Used in AI?
🤖 How Is This Used in AI?
Every production chatbot, AI assistant, or agent with any meaningful conversation length needs a deliberate context management strategy — this isn’t optional infrastructure, it’s a direct, practical consequence of context windows being finite and irrelevant context being actively unhelpful, not just wasted space.
9. When Should You Actively Manage Context?
- Any multi-turn conversation application, especially ones users might keep open for a long time
- Any system pulling in retrieved documents (RAG, Module 17) — including irrelevant retrieved content directly hurts answer quality
- Any time cost (Module 25) matters and context is growing unnecessarily large
10. When Is This Less of a Concern?
- Short, single-turn, or few-turn interactions that will never realistically approach a context limit
11. Common Mistakes
Incorrect idea
Assuming “just include everything” is the safe default.
Why it is incorrect
As shown directly, irrelevant context isn’t neutral — it can really hurt response quality, not just cost more.
Incorrect idea
Truncating without considering what’s actually important.
Why it is incorrect
Blindly dropping the oldest turns can lose really relevant information — sometimes a summary (Strategy 2) is a better fit than simple truncation.
Incorrect idea
Never revisiting the context strategy as usage patterns change.
Why it is incorrect
A strategy that worked for typical short conversations might fail badly for a small number of unusually long ones — worth monitoring, not setting once and forgetting.
12. Limitations
- Truncation and summarization both involve real trade-offs — some information is inevitably lost or compressed, and there’s no strategy that perfectly preserves everything while also staying within limits
- Retrieval-based strategies (Module 17) require real infrastructure (a way to search and pull back relevant content) — not a trivial addition to a simple chatbot
- There’s no single “right” context management strategy — the correct choice depends on the specific application’s needs, exactly the kind of judgment call this module can’t fully automate for you
Analogy: The Desk Workspace Cleanup Think of managing the context window like working at a small wooden office desk:
- The Messy Desk (Dumping everything): As you work, you stack every report, note, receipt, and textbook on the desk. Eventually, there is no physical space left to write a new page.
- Strategy 1: Truncation (Sweeping to the floor): You sweep the oldest stacks of paper off the edge of the desk directly onto the floor. You only keep the 5 most recent pages in front of you. (Fast, but you forget key details from last week).
- Strategy 2: Summarization (The Daily Digest): You read the pile of old reports, write a 1-paragraph summary index card, throw the thick reports in the recycle bin, and keep only the index card + the current active page on the desk. (Highly token-efficient, retains core highlights).
- Strategy 3: Retrieval/RAG (The Filing Cabinet): You put all reports in a metal filing cabinet in the hallway. You keep nothing on your desk. When someone asks about a specific vendor, you stand up, retrieve just that vendor’s invoice, place it on the desk, answer, and return it. (Keeps desk clean and scale infinite).
📊 Visual Chart: Context Management Strategies
Here is how the three core management workflows compare structurally:
graph TD
classDef main fill:#3498db,stroke:#333,stroke-width:1px,color:#fff;
classDef strat fill:#2ecc71,stroke:#333,stroke-width:1px,color:#fff;
ContextLimit["Context Limit Exceeded!"]:::main --> StrategySelection{"Select Strategy"}
StrategySelection -->|Simple conversation| Truncate["1. Truncation:<br>Drop oldest messages in array"]:::strat
StrategySelection -->|Long business session| Summarize["2. Summarization:<br>LLM condenses history into summary string"]:::strat
StrategySelection -->|Huge document database| Retrieve["3. Retrieval (RAG):<br>Fetch k-most similar snippets via vector search"]:::strat
Truncate --> Output["Re-assembled clean payload within token constraints"]
Summarize --> Output
Retrieve --> Output13. Quick Reference — The Whole Idea in One Diagram
context_used = system + history + retrieved content + current message
context_used <= context_limit
↓
If approaching the limit, or context includes irrelevant material:
↓
TRUNCATE (drop oldest) | SUMMARIZE (compress older turns) |
RETRIEVE (pull back only what's relevant, Module 17)14. Prompts in Code — Calling an LLM
Here’s how context management actually looks when calling an LLM through code — tracking token usage and applying a real strategy before it becomes a problem.
Example 1 — Simple
A naive conversation that just keeps appending messages forever, with no management at all — useful to see what NOT to do at scale.
import anthropic
client = anthropic.Anthropic()
messages = []
def chat(user_message: str) -> str:
messages.append({"role": "user", "content": user_message})
response = client.messages.create(
model="claude-sonnet-4-6", max_tokens=200, messages=messages,
)
reply = response.content[0].text
messages.append({"role": "assistant", "content": reply})
return reply
print(chat("What's a good marinade for chicken?"))
# messages list grows unbounded with every call -- no managementExample 2 — Intermediate
A simple truncation strategy: only the most recent N turns are kept.
import anthropic
client = anthropic.Anthropic()
messages = []
MAX_TURNS_KEPT = 10 # keep the last 10 user+assistant pairs
def chat(user_message: str) -> str:
messages.append({"role": "user", "content": user_message})
response = client.messages.create(
model="claude-sonnet-4-6", max_tokens=200, messages=messages,
)
reply = response.content[0].text
messages.append({"role": "assistant", "content": reply})
# Truncate: keep only the most recent turns
max_messages = MAX_TURNS_KEPT * 2
if len(messages) > max_messages:
messages[:] = messages[-max_messages:]
return reply
print(chat("What's a good marinade for chicken?"))Example 3 — Production Grade
A context manager that summarizes older turns instead of simply discarding them — preserving important context while still controlling growth, and using a real token-estimate check to decide when to act.
import anthropic
client = anthropic.Anthropic()
class ManagedConversation:
def __init__(self, system_prompt: str, summarize_after_turns: int = 10):
self.system_prompt = system_prompt
self.summarize_after_turns = summarize_after_turns
self.summary = ""
self.recent_messages = []
def _summarize_old_messages(self, old_messages: list) -> str:
transcript = "\\n".join(f"{m['role']}: {m['content']}" for m in old_messages)
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=150,
messages=[{"role": "user", "content":
f"Summarize the key facts from this conversation "
f"in 2-3 sentences, for use as context in a "
f"future conversation:\\n\\n{transcript}"}],
)
return response.content[0].text
def send(self, user_message: str) -> str:
self.recent_messages.append({"role": "user", "content": user_message})
system_with_summary = self.system_prompt
if self.summary:
system_with_summary += f"\\n\\nContext from earlier in this " \\
f"conversation: {self.summary}"
response = client.messages.create(
model="claude-sonnet-4-6", max_tokens=300,
system=system_with_summary, messages=self.recent_messages,
)
reply = response.content[0].text
self.recent_messages.append({"role": "assistant", "content": reply})
# If recent history has grown too large, summarize and reset
if len(self.recent_messages) > self.summarize_after_turns * 2:
new_summary = self._summarize_old_messages(self.recent_messages[:-2])
self.summary = (self.summary + " " + new_summary).strip()
self.recent_messages = self.recent_messages[-2:] # keep only latest exchange
return reply
chat = ManagedConversation(system_prompt="You are a helpful cooking assistant.")
print(chat.send("What's a good marinade for chicken?"))This directly implements Strategy 2 from Section 5: instead of losing older context entirely (truncation) or letting it grow unbounded, the conversation periodically compresses itself into a running summary — balancing relevant context retention against real, growing token cost.
When to use it—and when not to
Use it when:
- long conversations or document workflows.
- the model needs selected prior state.
Do not rely on it when:
- all history is copied forward without relevance checks.
- summarization may remove legally or operationally important details.
15. Interview Questions
Q: Why isn’t “just include as much context as possible” a safe default strategy?
Ans: Beyond the direct cost of more tokens, irrelevant context can actively hurt response quality — it can distract the model with information unrelated to the current question, or in some cases include outdated information that conflicts with more current details. The goal is relevant context specific to the current request, not maximum context regardless of relevance.
Q: What’s the difference between truncation and summarization as context management strategies, and when might you prefer one over the other?
Ans: Truncation simply drops the oldest messages once a limit is reached — simple to implement, but it can lose really relevant information from earlier in the conversation. Summarization compresses older messages into a shorter representation that preserves key facts while freeing up space. Summarization is generally preferable when early context still has some ongoing relevance to later parts of the conversation; simple truncation is fine when only the most recent exchanges really matter going forward.
Q: Why does a long-running chatbot conversation become increasingly expensive over time, even if the user’s individual messages stay short?
Ans: Because the full conversation history is resent with every new message (Module 15) — as the conversation grows turn by turn, the total token count sent with each request grows too, even though any single new message might be short. This is a direct, unavoidable consequence of how multi-turn conversations work with LLM APIs, unless an active context management strategy (truncation, summarization, or retrieval) is in place to control that growth.
Q: How does context management relate to RAG (retrieval-augmented generation)?
Ans: RAG is essentially a specific, powerful context management strategy — instead of keeping all potentially relevant information in the conversation directly, it’s stored externally and only the specifically relevant pieces are retrieved and inserted into the prompt for a given request. This avoids ever needing to include an entire knowledge base or full history in context at once, directly solving the same “too much potentially relevant material” problem this module addresses, just applied to external documents rather than conversation history.
16. What You Should Remember
- Context management means deliberately deciding what’s actually relevant, not including everything available — more context isn’t automatically better.
- Real strategies include truncation (simple, can lose information), summarization (preserves key facts, adds processing), and retrieval (pulls back only what’s relevant, Module 17).
- A long conversation’s cost grows because the full history is resent every turn — this is a direct, practical reason context management matters, not just a theoretical concern.
17. Quick Practice
For a fitness-tracking chatbot that users might chat with daily for months, would you lean toward truncation, summarization, or retrieval as the primary context strategy? Explain your reasoning.
18. Next Step
Next: Module 17 — Prompt Engineering for RAG — how prompting changes specifically when an AI receives retrieved documents alongside a user’s question, and the real, common mistakes in RAG prompt design.
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed