Most systems pick one model and send everything to it. That is the right way to start and a poor place to stay, because workloads are rarely uniform: classifying a support ticket and drafting a legal summary have very different requirements and very different costs.
Model routing means matching the request to the capability it actually needs. Done carefully it reduces cost substantially and improves latency on the majority of requests.
Segment the workload first
Before routing anything, look at what your system is actually asked to do and in what proportions. Almost every workload turns out to be dominated by a small number of simple, repetitive tasks, with a minority of genuinely hard ones.
That distribution is the opportunity. If four fifths of requests are classification or extraction, sending them all to a frontier model is paying a large premium for capability that does nothing on those tasks.
Route on something you can determine cheaply
- Task type, where the calling code already knows what it is asking for. This is the cleanest signal and it is free.
- Input length, since very long contexts genuinely need different handling.
- Customer tier or transaction value, where the cost of a poor answer varies.
- A small, fast classifier, when the task type is not known in advance.
Prefer the first wherever possible. Routing decided by the code that made the request is deterministic, testable and free, and it avoids adding a model call to decide which model to call.
Most of the saving comes from realising that four fifths of your traffic is classification, and classification has not needed an expensive model for a long time.
Hannah Berg, Lead AI Engineer, Engineered With AI
Escalate rather than guess
A useful pattern runs the cheap model first and escalates when the result fails a check: schema validation failed, confidence was low, the output was empty, or a deterministic rule was violated.
This works when the check is reliable and cheap. It stops working when escalation becomes the norm, at which point you are paying for both calls on most requests and would be better sending that category straight to the larger model.
Measure the escalation rate per category
The rate tells you whether the routing is right. A category escalating on most requests should be routed up permanently. One that almost never escalates could probably use something cheaper still, and both adjustments are invisible without the per-category breakdown.
A fallback has to be genuinely different
Retrying the same model on the same endpoint after a failure handles a transient blip and nothing else. When a provider has a real outage, that fallback fails with the request.
A meaningful fallback uses a different provider, or a different region, or degrades to something simpler that still serves the user. Which of those is right depends on whether a slightly worse answer is better than no answer, and that is a product decision rather than a technical one.
Test the fallback path deliberately
Fallbacks are written once and almost never exercised, which means they are frequently broken by the time they are needed: a stale credential, a schema the other provider does not accept, a prompt tuned for a different model.
Route a small share of traffic down the fallback path routinely, or test it on a schedule. A path that has not run in six months should be assumed broken until demonstrated otherwise.
Keep the routing visible
Log which model handled each request, and include it in the record alongside the prompt version. Without it, an investigation into why quality dropped has to guess at which path the request took.
It also makes the cost analysis possible, because spend per model per category is what tells you whether the routing is earning its complexity. This sits alongside the wider observability picture rather than separately from it.
Fallbacks change your data handling position
A second provider means a second set of terms about retention, training and processing location. A fallback that silently sends regulated data somewhere your contracts do not cover is a compliance problem that only appears during an incident.
Check the terms for every model routing destination and every fallback before wiring it in, and exclude categories that cannot leave a jurisdiction from the fallback path entirely.
Paying premium rates for simple tasks?
We will look at your workload and route it so cost and quality land where they should.
