Agent Routing

A hospital reception desk sends patients to different specialists. Routing performs the same job for agent requests: classify the need, then choose the appropriate path.

Request → router → specialist A / B / C → result

What You Will Learn

• How routing differs from delegation and orchestration.
• How rules, classifiers, or LLMs can select a route.
• How fallbacks, confidence thresholds, and evaluation make routing safer.

Module 5 covered the criteria for choosing which agent should handle a subtask — capability, cost, latency, reliability, permissions. This module covers something that happens earlier and separately: the actual mechanism that classifies an incoming request and dispatches it toward the right agent in the first place.

Delegation asks: given several candidates, which one should do this? Routing asks: which candidates should even be considered, before delegation’s criteria are applied at all?

Get routing wrong, and delegation never gets the chance to make a good decision — the request already went to the wrong place.

What the layered routing decision actually looks like

Incoming Request
     ↓
Rule-based check: does this match a known, explicit pattern?
     ↓ (no match)
Semantic check: does this score above the confidence threshold
against any candidate destination's embedding?
     ↓ (below threshold)
LLM-based decision: ambiguous case, worth the cost
of a real model call to classify correctly
     ↓
Route to selected destination
     ↓
(Module 5's delegation criteria now apply within that destination)

Notice each layer only activates when the one before it couldn’t confidently resolve the request. This is the actual mechanism behind the cost-and-latency numbers covered later in this module — most requests never reach the expensive final layer at all.

---

The single most important fact about routing failures

This is worth understanding before anything else in this module, because it changes how seriously you should treat routing accuracy:

“The most dangerous property of query routing is that errors are invisible. A misrouted query doesn’t throw an exception — it returns a confident, plausible, wrong answer from the wrong data source.” — Guild.ai, Query Routing (AI)

This is directly the same shape as Module 10’s silent race-condition corruption. Nothing crashes. Nothing logs an error. A request reaches an agent that was never the right one for it, and that agent — doing its best with a request outside its actual competence — produces something that reads as a real, confident answer. The downstream consumer of that answer has no structural signal telling them anything went wrong.

---

Three real strategies, and when each fits

Production routing in 2026 converges on three named approaches, each suited to a different kind of input.

Rule-based (logical) routing. Requests are dispatched using predefined conditions — keywords, structured fields, document type, known query templates. It’s cheap, deterministic, and easy to audit, because the entire decision path is explicit and inspectable. The real weakness: rules are brittle. They need active maintenance as the actual distribution of incoming requests shifts, and novel phrasing that doesn’t match any rule produces a real misroute. (Redis, LLM router architecture: best practices for 2026)

Semantic (embedding-based) routing. Incoming requests and candidate destinations are both encoded as vector embeddings, and the request routes to whichever destination has the highest similarity score. This fits open-ended natural language where meaning matters more than exact wording — “what’s 15% of 200?” has no obvious keyword match for a math agent, but its meaning clearly does. (nx1, AI Query Routing 2026)

Predictive (LLM-based) routing. A model itself estimates which destination will handle a given request best, weighing quality against cost. This is the most expensive option per request, and it’s reserved specifically for cases the first two strategies can’t confidently resolve.

The real architecture combines all three

Production systems rarely pick just one. The dominant real pattern: “Most production platforms combine the two and add an LLM-based decision only for the residual ambiguous cases, because paying for a model call on every request erases much of routing’s savings.” (nx1)

This is worth stating precisely as a layered fallback, not a menu of alternatives: rule-based routing handles the obvious cases fast and cheap; semantic routing catches what rules miss; an LLM-based decision is reserved for whatever’s left over, ambiguous, and worth the extra cost to get right.

---

Confidence thresholds: the load-bearing decision inside semantic routing

It’s worth knowing precisely why semantic routing alone isn’t sufficient, and what real systems do about it.

“Without a threshold, a semantic router can confidently misroute ambiguous traffic. The common pattern is to use the semantic router as a fast path and fall back to an LLM for queries that score below your threshold.” — Redis, LLM router architecture: best practices for 2026

A real, concrete implementation detail from a production tutorial: a confidence threshold of 0.55 cosine similarity — below that, the router doesn’t guess, it falls back to a general-purpose model rather than committing to a low-confidence match. (Building a Production-Grade Semantic Router, Medium)

This directly mirrors Module 10’s honest framing of race conditions: the danger isn’t that semantic routing is unreliable in general — it’s that a similarity score just above an arbitrary cutoff and a score just below it can represent similar confidence, and treating the cutoff as a hard, binary “correct” versus “wrong” signal is exactly what produces invisible misrouting.

Consider a concrete case: a request scoring 0. 56 against a “billing” agent’s embedding and 0. 54 against a “technical support” agent’s embedding routes to billing, full stop — the threshold treats a two-point difference as decisive, when in semantic terms the request may have been almost equally about both.

If the actual issue was a billing error caused by a technical bug, routing it exclusively to billing means the technical root cause never gets investigated, and nothing about the routing decision itself signals that anything was uncertain. This is precisely why production systems don’t treat “above threshold” as equivalent to “correct” — they treat it as “confident enough to proceed without the added cost of an LLM-based decision,” a different, more honest claim.

---

Real misrouting rates, precisely measured

This is worth knowing in actual numbers, not a vague sense that routing “sometimes” fails.

Red Hat’s own developer blog, describing their vLLM Semantic Router in production testing, reports the pretrained model achieved 80% accuracy on a four-tier classification task — meaning a 20% misrouting rate. Their own framing is direct: “One in five requests is sent to the wrong model. That is not a tuning issue. That is a system-level limitation.” (Red Hat Developer, Improve vLLM Semantic Router accuracy with fine-tuning)

Misrouting gets measurably worse as the option space grows

A 2026 research paper, Switchcraft, published an actual measured breakdown of misroute rate against the number of tool definitions a router had to choose among:

Tools available	Misroute rate
1	15.3%
2–3	17.2%
4–6	20.6%
7–10	23.2%
11–50	39.8%

(Switchcraft: AI Model Router for Agentic Tool Calling, arXiv)

Read this table carefully — it’s not a small trend. Misroute rate more than doubles between a single-tool router and one choosing among 11–50 tools. This is directly, precisely why Module 5’s capability-matching discussion and Module 2’s tool-description discipline matter as much as they do: every additional option a router has to distinguish between is a real, measured increase in failure rate, not a free expansion of capability.

---

Skill collision: routing failure with a name

This is worth knowing as its own concept, because it explains why the table above trends the way it does, not just that it does.

Skill collision occurs when semantically overlapping skill or agent descriptions compete for the same incoming query population. A 2026 research paper studying this in production found it’s “most acute during onboarding” — adding a new skill to an already-deployed system forces that skill’s description to be precisely positioned relative to every incumbent skill already in the router’s option space, or the new addition steals traffic that should have gone elsewhere. (A Single Rewrite Suffices, arXiv)

This is directly the same failure Module 5’s Grep versus Glob discussion warned about, now understood as a ongoing maintenance cost rather than a one-time design decision. Every time a system adds a new agent or skill, the router’s entire existing option space needs re-evaluation — not just the new addition’s own description in isolation.

---

What routing costs, and what it saves

It’s worth having real numbers for both sides of this trade, since Module 5 already covered adjacent latency figures for heavier ML classifiers specifically.

On the cost side: a well-optimized router — simple rule or embedding lookup, not a full LLM call — contributes as little as 10 to 50 microseconds of latency, negligible against the 500 to 2,000 milliseconds a real model inference call takes. (Guild.ai) This is a real, important reconciliation with Module 5’s own latency table: that module’s 50–100ms figure was specifically for heavier semantic and ML-classifier routing; a lightweight rule-based or cached-embedding lookup can be orders of magnitude faster still.

On the savings side: routing can reduce overall inference costs by 30 to 85%, by directing simple requests to smaller, cheaper models and reserving expensive frontier models specifically for complex ones. In RAG systems specifically, query routing has improved accuracy from 58% to 83% in real production deployments, by matching query types to the retrieval strategy actually suited to them rather than treating every question identically. (Guild.ai)

---

Applying this to the recurring scenario

The legal-contract pipeline currently uses a fixed structure — every contract goes through the same Planner-Executor-Critic sequence regardless of content. Introducing routing would mean something different: a lightweight classifier examining an incoming contract before the Planner even begins, directing routine, template-matching agreements down a fast, simpler path while routing contracts with unusual clauses or unfamiliar structure toward the full checklist process.

Run this module’s numbers against that idea honestly. A rule-based first pass — checking against known template signatures — would be fast and auditable for the common case. Anything that doesn’t match a known template falls to semantic routing, comparing the contract’s actual content against embeddings of previously-seen contract types. And a contract that scores below a real confidence threshold — novel, unfamiliar structure — escalates to the full Planner-driven review rather than risking a confident, silent misclassification of something that needed real scrutiny.

The Switchcraft data above is a direct, concrete warning for exactly this kind of extension: adding more contract-type categories to route between doesn’t just add capability, it measurably increases the misroute rate for every category, including the ones that already worked well before the expansion.

This is worth taking seriously as a constraint on system growth, not just a footnote. A team adding a fifth or sixth contract category to this routing layer should expect to re-test the existing categories’ accuracy, not just validate that the new category itself routes correctly — precisely because skill collision means every addition changes the shape of the entire decision space, not just the part that’s obviously new.

---

Interview-relevant framing

Q: Why is a misrouted request more dangerous than an outright routing failure?

Ans: Because an outright failure — a request nothing can handle, an explicit error — is visible and gets caught immediately. A misroute sends the request to a capable agent that simply isn’t the right one for this specific case, and that agent does its honest best, producing a confident, plausible answer that’s still wrong. Nothing in the system flags this as an error, because nothing technically failed.

Red Hat’s own production testing found a 20% misrouting rate in a real deployment — a large, silent failure surface if nobody’s specifically measuring for it.

Q: **How would you decide between rule-based, semantic, and LLM-based routing for a new system? **

Ans: I wouldn’t pick just one — production systems layer all three. Rule-based for the obvious, structured cases, since it’s cheap and fully auditable. Semantic routing for open-ended natural language where meaning matters more than exact phrasing, with a real confidence threshold so it falls back rather than guessing on ambiguous cases. And an LLM-based decision reserved specifically for whatever’s left over after the first two layers, since paying for a model call on every single request would erase most of routing’s actual cost savings.

A third question worth preparing for:

Q: Why does misrouting get worse as you add more agents or skills to a system, and what would you do about it?

Ans: Because every new option added to a router’s decision space is one more thing every existing option now has to be distinguished from — a real, measured effect, not a theoretical concern. A 2026 study found misroute rate climbing from about 15% with a single tool to nearly 40% with 11 to 50 tools available.

The real mitigation isn’t avoiding growth entirely, it’s treating each new addition’s description as something that needs positioning against every incumbent option, not just written in isolation — directly the same discipline Module 2 covered for distinguishing tools like Grep and Glob, now applied at the scale of an entire router’s option space.

Common Misconception

Incorrect idea: A router that sounds confident selected the right agent.

Why it is incorrect: Confidence is not correctness. Routes need labeled evaluation cases, thresholds, fallbacks, and monitoring for changing traffic.

Key takeaways

• Routing is the dispatch mechanism deciding which candidates a request even reaches; delegation (Module 5) is the criteria used to choose among those candidates once routing has already narrowed the field.
• Misrouted requests fail silently — a wrong destination still produces a confident, plausible answer, with nothing structurally flagging that anything went wrong, directly parallel to Module 10’s silent race-condition corruption.
• Three real strategies exist — rule-based, semantic, and LLM-based — and production systems layer all three, reserving the most expensive option for ambiguous residual cases only.
• Confidence thresholds are the load-bearing decision inside semantic routing — a real production implementation uses 0.55 cosine similarity as its cutoff, falling back rather than guessing below it.
• Real, measured misrouting rates are significant — Red Hat’s own production testing found a 20% misroute rate on a four-tier classification task.
• Misrouting gets measurably worse as the option space grows — a real study found misroute rate climbing from 15.3% with one tool to 39.8% with 11–50 tools, more than doubling.
• Skill collision — overlapping descriptions competing for the same query population — is most acute exactly when a system adds a new agent or skill, requiring the entire existing option space to be re-evaluated, not just the new addition in isolation.
• Lightweight routing adds negligible cost (10–50 microseconds for a well-optimized router) against real savings (30–85% cost reduction, and accuracy improvements from 58% to 83% in production RAG systems).

None of this replaces Module 5’s delegation criteria — routing and delegation are sequential, not competing concerns. A request that’s routed correctly still needs a real delegation decision once it arrives, and a request delegated well against the wrong candidate pool because routing failed first was never going to reach the right agent no matter how sound the delegation logic downstream turned out to be.

Module 12 moves from how requests get to the right agent to a concern every pattern in this course depends on regardless of routing or topology: shared state and memory — what agents need to see in common, what should stay private, and how state conflicts get resolved when they inevitably arise.