A circuit board of routes representing requests directed to different language models

LLM Model Routing by Task, Cost, and Quality

August 12, 2026 9 min read

Model routing sends routine requests to a small model and hard ones to a strong model. The bill drops, and quality can slip without anyone noticing: a tool call loses an argument, a schema breaks, a high-stakes approval lands on the weak model. An average success rate hides all of it until someone acts on a bad answer.

A prompt classifier alone is not a production router. You need deterministic capability gates, a threshold tested on your own traffic, a strong fallback, versioned decisions, and a release process that compares routed output with the unrouted result. You also need to know when not to route at all.

Ask first whether you need a router

A router predicts difficulty before the model runs. A cascade runs the cheap model first, checks the result, and escalates only when the check fails. They are different tools, and the choice depends on one question: can you verify an answer cheaply and deterministically?

If you can, as with schema-valid extraction, a tool call that must match a contract, or a draft that must cite a source, a cascade needs no prediction. The check decides. You pay for the weak model on every request and for the strong model only on the failures. If you cannot verify an answer cheaply, a router has to guess from the prompt alone, and every guess is a bet you cannot audit afterward.

A broken router shows up as a quality complaint, not an error, and it has happened at the largest scale. After GPT-5 launched, Sam Altman wrote that "the autoswitcher was out of commission for a chunk of the day, and the result was GPT-5 seemed way dumber", as TechCrunch reported. OpenAI also said it would adjust the decision boundary and make clearer which model answered. Three controls follow. Decide which model a request gets when the router fails, and make it the stronger one for work where a weak answer is costly. Alert on the share of traffic sent to each tier, since a router that quietly sends everything to the cheap model keeps every health check green. And expose the chosen model in your logs and, for users, in the interface, so a complaint can be tied to a route.

The cost of the cascade is latency on the hard cases, which pay for two calls. Accept that when a wrong answer costs more than a slow one.

Why routing fails in production

Routing looks simple for plain text. Internal workflows add constraints that a difficulty score cannot see: a specific tool schema, image input, a long context window, strict JSON, a regional endpoint, or a model approved for a data class. A cheap model can understand the prompt and still be ineligible. RouteLLM issue 63 records a case where a selected local model rejected a forwarded parameter it did not support. The routing decision worked and the request contract did not survive it.

Thresholds do not transfer. The RouteLLM documentation recommends calibrating with data that resembles your requests, and a threshold from general chat may not fit invoice extraction or policy interpretation. Issue 79 raises a related concern about scores tied to internally fixed model identities rather than the pair you deploy. Treat model names, versions, and routing scores as one versioned unit.

Providers add their own limits. Amazon Bedrock's intelligent prompt routing is documented as optimized only for English prompts, cannot adjust decisions from application-specific performance data, and routes between exactly two models in one family. If a large share of your traffic is in another language, the managed router is untested for it by the vendor's own account.

Mistakes are also not symmetric. Sending a simple summary to the strong model wastes money. Sending a payment approval to the weak model can cause a bad business action. The policy has to weigh the consequence of an error, not only the estimated difficulty.

Your calibration labels can be wrong in a predictable direction

Router evaluation usually asks an LLM judge which model answered better. A 2026 study of 206,000 query-model pairs across six benchmarks, Unsolvability Ceiling in Multi-LLM Routing, found that much of the apparent room for routing to improve is an artifact of evaluation. The authors point to judges that favor verbosity over correctness, answers truncated under fixed generation budgets, and output-format mismatches. They also report that standard routers collapsed to predicting the majority class, with about 79 percent of labels naming the smallest model as optimal.

Two lessons carry over to an internal router. If most of your labels say the small model is fine, a router can score well by always picking it, so evaluate on the hard class separately. And grade with deterministic checks (schema validity, exact match on the fields that matter, tool name and arguments) before you trust any judge model. The same paper recommends exact-match anchoring and dual-judge validation for this reason.

Define the routing contract before choosing a router

The application already knows the endpoint, requested output, available tools, caller, and business operation. Pass that context into the policy layer instead of asking a router to infer it from free text. For each route, record:

The models allowed for its data classification and region.
Required capabilities such as tool calling, vision, context size, and structured output.
The maximum acceptable latency and cost per request.
The consequence of a weak answer.
The evaluation set and metric used to judge the route.
The fallback model and the conditions that force it.

Hard checks decide eligibility first. The router then chooses among models that can safely run the request. Use the strong model as the default fallback. Unknown routes, missing capability records, unsupported locales, and malformed policies should all select it.

Use deterministic gates before learned scoring

choose_model(request, route_policy, model_catalog):
    candidates = models_allowed_for(request.data_class, request.region)
    candidates = candidates_with_all(route_policy.required_capabilities)

    if route_policy.risk == "high":
        return route_policy.strong_fallback, "high_risk"

    if request.context_tokens > route_policy.weak_context_limit:
        return route_policy.strong_fallback, "context_limit"

    if candidates does not include route_policy.weak_model:
        return route_policy.strong_fallback, "capability_mismatch"

    score = difficulty_router.predict(request.prompt, request.features)

    if score >= route_policy.strong_threshold:
        return route_policy.strong_fallback, "predicted_difficulty"

    return route_policy.weak_model, "eligible_low_difficulty"

Keep the model catalog in versioned configuration: capability flags, context limits, regions, data approvals, provider identifiers, and exact model versions. Any deployment that changes those values should trigger routing regression tests.

Force the strong model where a routing mistake is expensive: changing permissions, approving a payment, interpreting a contract clause for action, or generating a command that will run unreviewed. Learned scoring suits bounded, reversible work such as drafting, classification with human review, and summaries a user can inspect.

Calibrate the threshold on local tasks

The RouteLLM paper frames routing as a choice between a stronger and a weaker model under a cost and performance tradeoff, and uses a threshold to set the share of traffic each model gets. The right threshold is specific to your workload.

Build a labeled set from production-shaped requests, with sensitive values removed and the task mix, prompt length, tool use, languages, and failure cases kept. Run both models on every request and grade with the workflow's own checks. An extraction route grades schema validity and field accuracy. A support route grades category, escalation decision, and policy citations. A tool route verifies tool name, arguments, and authorization preconditions.

Compute a separate threshold for classes that differ materially. One global threshold lets abundant easy traffic swamp a small, hard class. For each class report weak and strong pass rates, the false-weak rate (router chose weak, only strong passed), strong-route share, cost per accepted result, tail latency, and fallback and execution-error rates.

Set the threshold against an explicit false-weak budget. If a workflow can tolerate one weak-model miss per thousand reviewed drafts, test to that target. If it tolerates none, take it out of learned routing and force the strong path.

Keep routing separate from execution fallback

The routed model can still fail while running: a provider rejects a parameter, quota runs out, a call times out, or the output is invalid. Keep an execution fallback after the routing step. Retry on the strong model for unsupported-parameter errors, invalid schema, tool contract failures, or a detectable low-confidence result. Do not replay a side-effecting request after an ambiguous timeout, because the first call may already have started a tool. Those workflows need an idempotency key and a durable record of whether execution began.

Log routing and fallback as separate events with the route ID, policy and router versions, candidate model versions, selected model, reason code, score band, fallback reason, latency, and cost. A fallback logged as an ordinary success hides degradation. Reason codes let you tell harder traffic from a broken capability catalog or a changed threshold when strong-model usage jumps.

Shadow the router before it controls traffic

Let the new router decide without enforcing. Keep serving the strong model, record what the policy would have chosen, and for a sample run the weak model asynchronously and grade both outputs. Shadow mode shows whether the live task mix matches the calibration set, whether new prompt shapes fall to the fallback, whether one tenant or language gets more weak routes, and whether the projected saving survives retries.

Canary only after false-weak rate, execution errors, model mix, and cost meet the release criteria, and keep a strong-model control group so routing effects are separable from prompt or retrieval changes. Tie rollback to a harm metric as well as spend: a release that hits the cost target while doubling schema repairs or human escalations has failed.

Detect drift after launch

Pin exact model versions where the provider allows it, and treat a changed alias as a new candidate that needs the paired evaluation again. Monitor by task class and policy version for shifts in strong-route share, fallback rate, invalid output, tool errors, and reviewer rejection. Sample accepted weak-model results for delayed review, since structural checks miss some defects.

Recalibrate after prompt changes, new tools, another language, a model upgrade, or a sustained change in request mix. Do not train only on cases the router selected: once the policy avoids weak-model failures, those failures vanish from its data. Keep a randomized evaluation sample across both routes.

When not to route

Routing adds a model catalog, a labeled set, shadow runs, and a rollback switch, all of which need an owner. If the strong model's monthly spend on a route is small next to the engineering time to maintain that machinery, use one model. If a route is rare and high-stakes, force the strong model. If your only cheap signal is the prompt text and a wrong answer is costly, route nothing until you can write a check.

Verify the complete routing path

Test the router inside the workflow, not as an isolated classifier. Fixtures should cover easy and hard requests, every required capability, oversized context, unsupported locales, high-risk operations, malformed policy, an unavailable weak model, invalid structured output, provider timeouts, and ambiguous side effects. Assert the selected model, reason code, fallback behavior, final result, and telemetry record for each. Then check that:

Every route has an owner, policy version, strong fallback, and task-specific evaluation.
Every model in the catalog has tested capability metadata and an exact provider identifier.
The false-weak rate stays within its class budget.
Unsupported capabilities always force an eligible model before learned scoring.
Execution fallback cannot duplicate side effects.
Shadow and canary results include a strong-model control.
One switch disables learned routing without changing callers.
Model, prompt, tool, and router upgrades trigger regression evaluation.

Start with one low-risk, high-volume route. Build fifty to a few hundred representative fixtures, run both models, set a false-weak budget, and shadow the decision. Add the next route only after the first has a measured quality result, a working fallback, and a tested rollback.

References

RouteLLM issue 63: a practitioner report about an unsupported parameter reaching a selected local model.
RouteLLM repository: implementation guidance for threshold calibration, serving, and router evaluation.
RouteLLM issue 79: a practitioner report about routing scores and the deployed model pair.
Amazon Bedrock intelligent prompt routing: routing criteria, fallback models, and documented limitations including English-only optimization.
Unsolvability Ceiling in Multi-LLM Routing: evidence that router headroom estimates are inflated by evaluation artifacts.
RouteLLM paper: primary research on learned routing between stronger and weaker models under a cost and performance objective.
TechCrunch, Sam Altman addresses bumpy GPT-5 rollout for the autoswitcher outage and the planned decision-boundary change.

About Fire In Belly: Independent senior engineering from Tallinn, Estonia. We design and build AI workflow automation with the capability gates, calibrated thresholds, and shadow-tested rollout described above, at published fixed prices. Schedule a call to discuss your next project.