Routing Each Request to the Right Model

Many requests a product receives are simple. A reminder, a reformatted address or a one-line summary does not need the largest model available, and sending everything to that model costs more and waits longer than it should. A routing layer decides which model handles each request. Done well, it saves money and keeps quality where it matters. Done badly, it sends hard cases to a weaker model, and nobody notices until a user does.

Why routing can pay

Models differ in cost, speed and capability, and a routing layer exploits that spread. If a share of your traffic can be answered well by a smaller or cheaper model, the saving on that share can be large. The benefit depends entirely on your traffic. Measure the distribution of your requests before designing anything. If most of them are of one kind, a single well-chosen model may be simpler and safer than a router.

Count the router's own costs as well. The classifier takes time and sometimes money, its code needs tests, and every routing rule is another place where a change can go unnoticed. The saving has to be large enough to cover that overhead, and it is worth estimating the overhead before building anything.

Ways to classify difficulty

Three approaches are common. Rules look at visible properties: input length, the presence of code, a category field set by the interface, or particular keywords. They are transparent and cheap to run, but they miss difficulty that does not show on the surface. A small classifier trained on labeled examples from your own requests can learn subtler patterns. It needs labeled data and ongoing maintenance. A model asked to rate difficulty is flexible and quick to start with, but its rating is itself a model output, with its own errors and its own cost.

Whichever you choose, test the classifier on labeled examples it has not seen. Signals that often feed a router include:

• the length of the input and the number of distinct instructions in it
• the task type the interface declares
• whether the answer must follow a strict format or call a tool
• whether the request depends on content in the user's own documents
• the stakes of the action, such as sending a message versus drafting one
• a confidence signal from the cheaper model's output, where one is available

Treat each of these as a proxy. None of them measures difficulty directly, and several will be wrong in cases you care about.

Cheap first, then escalate

A common pattern sends the request to the cheaper model first and then checks the result. The check might be schema validation, a grader that scores the answer against a rubric, or a simple rule such as the cited document ID existing in your system. If the check fails, the request escalates to the stronger model. This pattern is sturdy because the checks decide, not the classifier, so a misjudged request still gets caught at the output.

The costs are time and complexity. An escalated request takes longer than one sent directly, and two calls cost more than one. Count the escalation rate. If a large share of requests escalate, the first model is not saving you much, and the design should be reconsidered.

Rare categories need care in both directions. A category with only a handful of requests cannot show a reliable difference, so either group it with a neighboring category or keep it on the stronger model until there is more evidence. Routing a category on thin evidence is a bet that can fail without anyone noticing.

Measuring the quality gap

Compare the routed system with a single strong model on a sample of real requests. Have the answers reviewed blind, so the reviewer does not know which model produced each one. Report results by category. An average can hide a serious failure in one area, such as a small model that handles simple questions well but fails on questions involving negation or dates.

Decide in advance how much quality difference is acceptable. Some categories may tolerate a small one. Others, such as anything involving money or medical information, may need none. Writing that threshold down before the test prevents the result from being interpreted to suit the outcome.

Caching and where routing breaks

Caching repeated requests is often simpler and more valuable than routing. Exact-match caching returns a stored answer for an identical input, but only while that answer remains valid, which is not always the case. Semantic caching reuses answers for similar questions, and it can return an answer to a question that differs in a detail that matters.

Routing breaks down in predictable ways. Misclassified hard requests reach the weak model silently. Escalation can double the wait for a share of users. Behavior that differs from one request to the next makes bug reports hard to reproduce. And the router is itself software that needs tests, monitoring and updates whenever the models change. Log which model answered each request, so these problems can be traced when they appear.

Practice: plan a routing experiment

Take a log of recent real requests, or collect a sample if you do not have one. Label each request by category without looking at any model output, so your labels are not influenced by the answers. Choose one cheaper model and one stronger model, and run each sample through both. Have the answers judged blind.

For each category, decide whether the cheaper answer is good enough, and if so, write the rule that would route it there. Finish by naming the one kind of request you would still send to the stronger model regardless of cost, and writing down why. That final judgment is the part of routing that no classifier can make for you.