Scaling Small Model Inference via Shared Queuing

The Breakthrough

Replacing top-down request routing with a centralized shared-queue architecture allows individual workers to pull requests and form their own batches, which effectively doubles cluster throughput by eliminating the bottleneck of stale worker-state visibility.

What Actually Worked

• Decoupled Queueing: Use NATS Jetstream as a centralized message broker to hold incoming requests, allowing workers to pull tasks independently rather than relying on a router to push data.
• Binary Serialization: Replace JSON-based API payloads with MessagePack to reduce overhead and enable efficient transmission of multi-modal data directly through the gateway.
• Rust Sidecar Abstraction: Implement a Rust sidecar to manage diverse model runtimes (PyTorch, Candle, SGLang), allowing the system to switch between runtimes without rewriting application logic.
• Local Queueing Optimization: For multi-GPU machines, implement a local queueing layer that allows processes to negotiate batch sizes locally, avoiding the latency of network hops for batch adjustments.
• Lazy Model Loading: Instead of pre-loading all models, implement a memory-pressure-based eviction and loading strategy to pack multiple small models onto the same GPU dynamically.

Context

Small models (those fitting on older, mid-range GPUs) are increasingly reaching frontier-level performance for specific tasks, such as Vietnamese receipt OCR or contract review. However, serving a wide fleet of these models breaks traditional infrastructure. Top-down routers fail to saturate workers because they cannot maintain an accurate view of the KV cache state across many small, fast requests. Furthermore, the constant need to deploy new LoRAs and fine-tunes creates a communication bottleneck between AI engineers and infrastructure teams. The proposed architecture shifts the complexity from the router to the worker, enabling a more flexible, self-optimizing cluster.

Notable Quotes

• "If you use an open source model that's trained to do OCR on receipts in Vietnamese, that project has seen the most receipts in Vietnamese... on that task that model will outperform pretty much anything else."
• "This conversation between the AI engineer and the infrastructure person in getting those LoRAs up there... that's the thing that takes time and basically that's like the main killer in organizational velocity."

Content References

{"type": "tool", "title": "NATS Jetstream", "url": "https://nats.io/", "context": "recommended"}, {"type": "tool", "title": "SGLang", "url": "https://github.com/sgl-project/sglang", "context": "mentioned"}, {"type": "tool", "title": "Candle", "url": "https://github.com/huggingface/candle", "context": "mentioned"}