The State of Model Routing: Beyond Naive Task Delegation

The Failure of Naive Routing

The panel argues that traditional 'task-based' routing—sending a specific prompt to the model that benchmarks best for that category—is fundamentally fragile. Because agentic workflows are dynamic (e.g., a session starting as a codebase query, shifting to feature implementation, and ending in debugging), a model chosen at the start often becomes a bottleneck. When a small model is pushed outside its training distribution, it tends to 'thrash'—executing inefficient tool-use loops that ultimately cost more than simply using a frontier model from the start.

The 'Sidekick' Architecture vs. Sub-Agents

Cognition’s approach, as described by Walden Yan, moves away from spawning multiple sub-agents in favor of a single 'sidekick' model. This architecture keeps the KV cache warm, significantly reducing costs by avoiding the need to re-provide context. By allowing a frontier model to act as the planner while delegating implementation to a sidekick, developers can achieve 'frontier-level' intelligence at a 40% lower cost. The panel emphasizes that compaction should be used to improve intelligence rather than just save costs, as aggressive context truncation often leads to performance cliffs.

Jagged Capabilities and Collaborative Training

There is consensus that no single model wins every task; models have 'jagged capabilities' based on their training corpora. Routing is therefore an exercise in understanding these behavioral nuances. Alex Atallah (OpenRouter) notes that routing systems were historically underutilized until specific use cases—like apps sending periodic heartbeats—demonstrated that a single application often has two distinct intelligence needs. The panel suggests that the future lies in training models specifically for collaboration, using RL to teach models how to act as effective orchestrators or reliable delegates.

Key Takeaways

• Avoid naive routing: Do not route based on static task types; route based on the evolving needs of a session.
• Use a 'sidekick' model: Maintain a single, continuous context rather than spawning multiple sub-agents to maximize KV cache efficiency.
• Frontier models as planners: Keep a high-intelligence model in the loop to monitor progress and detect when a delegate model is out of its depth.
• Optimize for distribution: Small models are highly cost-effective for in-distribution tasks but become liabilities when forced to handle out-of-distribution complexity.
• Prioritize cache warmth: The economic advantage of small models is often negated if they cannot leverage cached tokens effectively.