The inexpensive route produces the expensive correction
A short request goes to the cheaper model because the router assumes short means simple. The answer sounds clear but misses a condition that changes the customer's eligibility. A colleague later repairs the result manually. The model bill looks better; the workflow has become worse.
Routing is a product decision about which tasks receive which capabilities. It needs an explicit quality boundary, not just a preference for the least expensive available response. Input length may be convenient to measure, but convenience does not make it a useful proxy for difficulty or consequence.
For a hypothetical service assistant, finding an opening time and interpreting an exception to an account policy can arrive as equally brief questions. They should not inherit the same route without evidence.
Separate difficulty from consequence
A task can be easy to phrase and costly to get wrong. Define the action and the consequence before deciding how to process it. A suggested label for internal review has a different risk from an instruction that changes a customer record.
Our preferred routing brief describes task families, the evidence each needs and the point where uncertainty must leave automation. Some tasks may need a stronger model. Others need a deterministic lookup, a permission check or a human decision rather than more model capacity.
Keep those options visible. A routing architecture that can only choose between models tends to treat every failure as insufficient intelligence. Missing data and missing authority do not disappear when the model changes.
Evaluate the router as part of the system
A candidate model can pass a test set and still fail in a routed application because the wrong tasks reach it. Evaluate the combined path: classification, selected model, supporting tools and final outcome. Record the intended route as well as the actual route for representative examples.
Google's evaluation guidance starts from a defined evaluation goal and task-specific criteria, rather than assuming a generic quality score answers every question. Its metric guidance describes that approach.
For the service example, compare whether the customer received a correct next step with the required supporting evidence. Review the groups separately: routine lookups, ambiguous requests, missing information and consequential actions. An average can conceal a regression in the group that matters most.
A fallback is another route to test
If the preferred model becomes unavailable, the fallback should not silently accept responsibilities it was never approved to handle. The application can defer the task, restrict its action or request review. Availability is important, but returning an unsupported decision is not the same as remaining useful.
Define the fallback with the same clarity as the normal path. Explain whether it preserves the complete task or offers a reduced service. Record why it was chosen, so an incident investigation can distinguish a deliberate route from an emergency substitution.
Also count the work after the first response. Retries, escalation and correction belong to the comparison. A low initial inference cost does not prove that the full route is economical, and a slower route may be acceptable when it avoids repeated intervention.
Compare complete journeys this week
Choose an existing workflow and build representative examples from approved, appropriately handled material. Include cases that look simple but carry important conditions. Assign expected outcomes and permitted actions before comparing routes.
Run the current path, proposed route and fallback against the same examples. Inspect wrong routing decisions separately from wrong model answers. Then make the rollout reversible and review new failures against those categories. Change the route when the completed work supports the decision, not merely when a pricing calculation looks attractive.