From Router to Conductor

Ask a router which model should answer a prompt and it returns a fast, defensible answer. Ask an end-user which model they want and you have already lost them. That gap is the whole story. Model routing is a good idea that optimizes a decision most people using the product never wanted to make.

Chris Alexiuk, a developer advocate at NVIDIA, has argued that routing will quickly grow into orchestration. Routing earns its place inside a pipeline, where an engineer is happy to trade a little quality for a lot of cost savings. It becomes awkward the moment a person is on the other end of it, because a person is not holding a cost-quality dial. They are holding a goal. I think he is right, and the reason is worth spelling out, because it changes what you should build next.

What routing actually solves

A router looks at an incoming request and picks which model handles it. Send the easy questions to a small, cheap model and reserve the expensive frontier model for the hard ones. Two research systems made this concrete. FrugalGPT showed that a cascade of models, cheap ones first, could match the accuracy of a single large model at a fraction of the cost. RouteLLM trained routers on human preference data and cut costs by more than 80 percent on some benchmarks while keeping most of the quality.

The math is honest and the wins are real. If you run millions of requests, routing turns a runaway inference bill into a manageable one. Inside a pipeline you control, where the input distribution is known and the output feeds another program, a router is close to free money.

Routing answers one question: which model should handle this prompt? Orchestration answers a larger one: what has to happen for this goal to be met? The second question contains the first.

That containment is the key. A router is a single decision made once, at the front door, before any work has been done. It is a classifier. It has no memory of what happened last time, no ability to check its own guess, and no second move if the guess was wrong.

Where routing gets awkward

Put a router in front of an end-user and three problems show up at once.

First, it optimizes a metric the user does not hold. The user wants the task done well. The router is tuned to spend as little as possible while staying above a quality floor. Most of the time those goals agree. When they diverge, the router quietly picks the cheaper option and the user gets a worse answer they cannot see the reason for.

Second, it is a single point with no recovery. The routing decision happens before generation. If it sends a genuinely hard request to the small model, there is no step that notices the answer is weak and escalates. The mistake is baked in at the first move.

Third, it exposes a choice the user did not ask to make. Routing frameworks often surface model names, tiers, and toggles. That is plumbing leaking into the interface. Someone trying to draft a contract or debug a script does not want a menu of model options. They want the contract and the working script.

Orchestration contains routing

Orchestration keeps a goal in view and drives a sequence of steps toward it. Anthropic describes one common shape as orchestrator-workers: a central model breaks a task into subtasks, hands each to a worker, and combines the results. The controller holds state, so it knows what it has tried. It can call tools, run code, search, and read back the output. Most importantly, it can check the result of a step and decide the next one based on what actually happened.

Inside that loop, choosing which model runs a given subtask is still a real decision. It is just no longer the top-level decision. It becomes one node, wrapped by steps that can catch and repair its mistakes. Route a subtask to a small model, verify the output, and escalate to a larger model when the check fails. Routing survives, demoted from the interface to an implementation detail.

The ergonomics are the point

Routing is awkward for end-users for the same reason a car's transmission is awkward as a user interface. The transmission is a genuine engineering achievement, and drivers are glad it exists. They do not want to operate it gear by gear. Automatic transmission won not because it shifted better than a skilled driver, but because it moved the decision out of the driver's hands entirely. The driver states intent with the accelerator, and the machine handles the mechanism.

Orchestration is the automatic transmission for models. The user states a goal. The system decides how many steps it takes, which models and tools to use, and when the work is good enough to return. Routing is one gear-change inside that machine, and like a gear-change it should be invisible when it works. This is why routing reads as a stepping stone. It is the moment the field learned that not every request needs the biggest model. Orchestration is what you build once you accept that the user should never have to know which model ran at all.

What this looks like in practice

The migration path is incremental. You do not throw away the router. You wrap it.

Start with the router you already have and put a verification step after it. When a cheap model handles a request, check the output against the goal, whether that is a test suite, a schema, a rubric, or a second model asked to critique the first. If the check fails, escalate. That single change converts a one-shot classifier into a two-step loop and removes the worst failure mode, the confidently wrong cheap answer that nobody caught.

From there, give the controller memory and tools, so it can break a request into subtasks, run each, and assemble the result. Keep model selection where it belongs, buried a layer below the interface. The person on the other end should see a goal going in and a finished result coming out, with the machinery of which-model-when handled for them.

What leaders should do

1. Treat any model picker exposed to end-users as technical debt. If your product asks a customer to choose a model or a quality tier, put a plan in place to hide that choice behind a goal-level interface.
2. Add a verification step after every routing decision. Before adopting more models, make sure a weak answer from a cheap model gets caught and escalated. One check after the router removes the most damaging failure and costs little to build.
3. Fund the orchestration layer as a product surface, not as glue code. The controller that holds the goal, state, and tool access is where user experience now lives. Staff it and measure it like the interface it is.
4. Buy routing, build orchestration. Per-query model selection is becoming a commodity inside gateways and inference platforms. The loop that turns a goal into a verified result is where your product is different, so keep that in-house.