Most AI Work Can Wait: Routing Before Models
Most teams building agents start by picking the model, then design the architecture around it. That sequence is backwards. The article argues the real bottleneck isn’t model capability but routing — deciding which tier of model handles each request. By treating model selection as the last decision, teams can route 70-80% of traffic to cheap local or async models. Brian Armstrong’s observation about Coinbase cutting AI spend in half while token usage grew reinforces the same point: better defaults, routing, and caching matter more than choosing the best single model upfront.
The concrete technical path separates the routing problem into three distinct layers. A skill classifier turns raw user requests into concrete operations like “draft a reply” or “summarize a repo.” A router decides which tier executes that operation based on complexity, context size, and historical success rate — it reads the classifier’s label, not the prompt directly. A model selector then picks the cheapest model within that tier meeting a confidence threshold. The classifier is a language problem; the router is a scheduling problem. Conflating them buries model choice inside prompts and prevents A/B testing. Two feedback mechanisms sit on top of the router: synchronous failure-mode signals that annotate incoming routes with five features (missing context, long dependency chains, risky migrations, security-sensitive prompts, high-consequence writes), and nightly closed-loop feedback from batch evaluators that update router weights using async inference.
The practical takeaway is that surprisingly little agent work needs real-time answers once the system can queue. Draft replies, repo summaries, diligence memos, and nightly evaluator runs don’t need sub-second responses. Async batch reasoning operates two orders of magnitude cheaper than real-time inference. Once skill distillation flattens the operation set, 70-80% of agent traffic can run on local models for most non-coding work. Builders should design their system around routing, not models — and pick models last.
