The default agent architecture sends every request to the most capable model available. It is the simplest configuration and it produces the best quality, and it is also the most expensive way to operate — often by a large multiple, because a substantial fraction of agent requests are simple enough that a much cheaper model handles them identically.
Routing is the discipline of matching request to model. Done well it cuts cost significantly with no perceptible quality change. Done carelessly it produces worse quality at higher cost, which is a genuinely achievable outcome and worth understanding before you start.
Route by Task Class First
The highest-value routing strategy is also the simplest: classify by what the request is, not by predicting difficulty.
An agent typically performs a small number of distinct operations. Classification, extraction, and formatting are usually well within a small model's capability. Planning, multi-step reasoning, and synthesis usually are not. Summarization sits in between depending on length and required fidelity.
Route on the operation, determined by your own code — which step of the workflow this is — rather than by inspecting the content. Properties that make this the right starting point:
Deterministic. The same operation always goes to the same model, so behavior is reproducible and debuggable.
Explainable. You can state the rule, and an auditor or a teammate can verify it.
Cheap. No classifier call, no latency added.
Measurable. You can benchmark each task class independently and know where the quality margin is thin.
In practice, per-step routing inside a multi-step agent captures most of the available saving. An agent whose planning step uses a frontier model and whose extraction and formatting steps use a small one costs a fraction of one that uses the frontier model throughout, with no measurable quality difference on the cheap steps.
Cascade Routing and Its Break-Even
Cascade routing tries the cheap model first, evaluates the result, and escalates to a stronger model if the result is inadequate. It sounds strictly better and it has a break-even point people rarely compute.
If the cheap tier succeeds a fraction p of the time, the expected cost is the cheap call always plus the expensive call a fraction (1-p) of the time, plus the evaluation cost. When p is low, you pay for both models on most requests and add latency. The rough shape: with a 10x price ratio between tiers, cascading is a saving when p is above roughly 0.7, and a loss below.
Two additional costs are easy to overlook. Evaluation is not free — if you use a model call to judge the cheap tier's output, that call has to be cheaper than the escalation it avoids, which often means it must be a deterministic check rather than a model call. Latency is additive on escalation, so p95 latency gets worse even when average cost improves.
Cascade works well where the success check is deterministic and free: schema validation, a required field being present, a numeric answer being in range, a tool call being well-formed. It works badly where "was this good enough" requires judgment. Where you need judgment, task-class routing is usually the better instrument.
Measure Cost Per Successful Task
The metric that keeps routing decisions honest is cost per successful task, not cost per call.
A cheap model that fails and triggers a retry, an escalation, or a second turn of clarification can easily cost more than one call to a capable model. Agent loops amplify this: a weaker model that takes seven steps instead of four costs more than the price ratio suggests, because context accumulates on every step and token consumption grows superlinearly with trajectory length.
So the comparison you want is not "model A costs a fifth of model B per million tokens." It is "on this task class, model A costs X per successful completion at this success rate and this step count; model B costs Y." That number sometimes favors the expensive model even on tasks where the cheap one mostly works. See agent evaluations: catching quality regressions and benchmarking agents.
Also track refusal and over-refusal rates per tier. A cheaper model that refuses legitimate requests more often generates retries and user frustration that no cost model captures.
Routing Is a Governance Control
Cost is the usual motivation for routing, and the governance uses are at least as valuable.
Capability ceilings on untrusted surfaces. A public-facing agent does not need frontier-model access with a 200k context window. Capping the model tier and context length reachable from untrusted input is one of the highest-value single controls against denial of wallet, because it bounds the worst-case cost per request.
Data residency and provider selection. Routing determines which provider — and therefore which jurisdiction and which contractual terms — processes the request. Data classified as restricted can be routed to a provider or a self-hosted model that satisfies its residency requirement, while general traffic uses the cheapest suitable option. This makes residency an enforceable routing rule rather than a policy document. See data residency for AI agents.
Sensitive-data routing. Requests whose content triggers a sensitivity classification can be routed to a model with stricter retention terms, or to a self-hosted model, or blocked. Combined with redaction, this is how regulated workloads use commercial models at all. See PII detection and redaction in AI pipelines.
Failover and availability. Provider outages happen. A routing layer with a configured fallback keeps the agent running on a second provider, and that is a policy decision rather than a code change — provided your prompts are not so tuned to one model that the fallback is unusable, which is worth testing rather than assuming.
Tenant-specific routing. A customer who requires their traffic on their own provider account, or a specific model, or a specific region. Only tractable when routing is configuration. See BYOK: managing LLM configurations.
Operational Requirements
Pin model versions. Floating aliases mean the model changes without notice, and your routing table's assumptions silently expire. Pin explicitly, and treat a version change as a change requiring re-benchmarking.
Re-benchmark on a schedule. Price and capability both move. A routing table set six months ago is probably leaving money on the table or has quietly degraded.
Keep the routing table as configuration, not code. Changing which model serves a task class should not require a deployment, and it should be versioned and auditable when it changes. See policy as code and guardrail versioning.
Record the routing decision. Which model served the request and why. Without it, cost analysis and quality investigation both stall — and a model mix shifting toward expensive tiers is one of the more useful spend anomaly signals.
Test prompts across tiers. A prompt tuned for one model often underperforms on another. If a fallback or a cheap tier is in your routing table, its prompts need their own evaluation rather than inheriting the primary tier's.
Common questions
Should I use a model-based router that predicts difficulty?
Only after task-class routing is in place and you have measured what remains. A learned router adds a call, adds latency, adds a component to maintain, and is opaque when it makes a bad choice. Task-class routing is free and captures most of the benefit. Where a learned router genuinely helps is within a single task class with wide difficulty variance — and you should be able to demonstrate that variance before building it.
How much can routing actually save?
It depends entirely on your task mix, and the honest answer is that you cannot know without measuring. Workloads dominated by extraction and formatting see large savings; workloads that are mostly complex reasoning see little. The way to find out is to benchmark each task class on a cheaper tier and compute cost per successful task — a day of work that either identifies a substantial saving or tells you routing is not your lever.
Does self-hosting change the calculus?
It changes the cost structure from per-token to per-GPU-hour, which favors high, steady volume and penalizes bursty low volume. Routing then becomes a capacity question: keep self-hosted capacity busy with the steady baseline and burst to a commercial provider for peaks. The governance benefits — residency, retention, no third-party data flow — are often the stronger motivation than cost.
What about routing between agents rather than models?
Same principle at a different layer, and it matters for the same reasons: a simple request handled by a lightweight agent with a narrow tool set is cheaper and safer than routing everything through a general orchestrator with broad authority. Routing by task class at the agent level is also a scope-reduction control, since the narrow agent has fewer connections. See orchestration patterns for multi-agent systems.
How Praesidia approaches routing
Praesidia holds provider credentials and LLM configurations at the gateway, so model selection is configuration rather than something embedded in each agent — routing rules, fallbacks, and tenant-specific provider assignments can change without redeploying agents. Because calls pass through the gateway, model tier and context limits can be capped per connection, which bounds what an untrusted surface can reach.
Every call records the model and version served, the token counts, the applied rate, and the cost with full attribution, so cost per successful task is computable per task class and a shift in model mix is visible as a spend signal. Tenants can supply their own provider keys and configurations where residency or contractual terms require it. See cost control for LLM applications and the Praesidia documentation.