llama-swap: One Endpoint for Hot-Swapping Local LLMs
Better Stackgo watch the original →
the gist
llama-swap proxies a single OpenAI-compatible API to multiple local LLM backends; it auto-starts requested models, forwards requests, unloads idle ones via TTL to free VRAM, and skips port/URL changes.
Proxy Mechanics
llama-swap runs as a single binary that exposes one stable OpenAI-compatible (and Anthropic-compatible) API endpoint. Clients send standard chat requests with a model field like 'quen-coder' or 'small-lm2'. llama-swap checks if the backend for that model runs; it forwards the request if ready, starts the backend if idle, or stops a conflicting model first. Tools like Open WebUI, Cursor, or custom agents use the fixed base URL without restarts or edits.
YAML Configuration Details
Users write a YAML config file defining each model. Each entry specifies a command to launch the backend (e.g., llama.cpp server with custom flags, GPU layers, context size), the model file path, and a TTL for idle unloading to reclaim VRAM. The demo shows a config with two entries: one for 'quen-coder' (coding model) and one for 'small-lm2' (fast chat model). llama-swap handles health checks before proxying.
Tradeoffs vs Alternatives
llama-swap prioritizes control over llama.cpp flags, custom backends (vLLM, Tabby API), and server-first setup for dev boxes, Docker, or homelabs; it lacks Ollama's model downloader/CLI or LM Studio's GUI. Setup demands model files, backend knowledge, and YAML tweaks, making it intensive but ideal for precise workflows with agents or scripts needing one endpoint.