Implementing Local LLM Routing with Nvidia Switchyard

The Breakthrough

By implementing a routing layer using Nvidia Switchyard, developers can dynamically delegate tasks between efficient local models like Nemotron 3.5 Lightning and frontier models like Claude 3.5 Sonnet, significantly reducing inference costs and latency while maintaining performance.

Routing Strategies and Implementation

Switchyard provides four distinct strategies to manage model selection, each with different cost and performance trade-offs:

• Random: Splits traffic on a fixed ratio, primarily used for A/B testing or establishing performance baselines.
• LLM Classifier: Uses a secondary model to read the incoming request and select the appropriate tier, though this adds latency and cost per query.
• Stage Router: Evaluates signals already present in the conversation, such as tool call failures or edit history, to determine the route without incurring extra inference costs.
• Escalation: Starts all tasks on a cheaper, local model and uses a judge to monitor for failure or complexity, escalating to a frontier model only when necessary.

To implement this, define a configuration profile in Switchyard that maps targets (e.g., 'capable' vs 'efficient') to specific API providers or local endpoints. For privacy-sensitive workflows, use a local model to digest raw data and redact PII before routing the sanitized payload to a frontier model for complex reasoning.

Performance and Throughput

Nvidia Nemotron 3.5 Lightning is a 30B parameter Mixture-of-Experts (MOE) model optimized for high throughput. It utilizes a hybrid Mamba-Transformer architecture to maintain a 1M context window and supports speculative decoding. In testing on a DGX Spark, the model achieved 71 tokens per second with a 70% acceptance rate for multi-token predictions. When compared against Claude 3.5 Sonnet for a 1400-token reasoning task, the local model completed the task in 20 seconds, compared to 50 seconds for the API-based model.