Running Claude Code with Local Models via Ollama
Better Stackgo watch the original →
the gist
You can replace proprietary LLMs in Claude Code by pointing its base URL to a local Ollama server, enabling the use of open-source models like Qwen or Gemma for zero-cost inference.
Redirecting Claude Code to Local Endpoints
Claude Code and similar CLI agents can be configured to use local models by modifying environment variables to point to an Ollama server. By setting the BASE_URL to your local Ollama instance (typically http://localhost:11434/v1), the agent interacts with open-source models as if they were standard API endpoints. This bypasses Anthropic's billing while maintaining the agent's existing workflow and file-system capabilities.
Hardware Requirements and Performance
Local model performance is strictly gated by available VRAM, which determines the maximum context window size. Systems with less than 24 GB of VRAM are limited to 4K context, while 28 to 48 GB of VRAM supports 32K context, and systems exceeding 48 GB can handle up to 256K context. On Apple Silicon, Ollama leverages the MLX framework to utilize unified memory and neural accelerators, which improves time-to-first-token and overall generation speed.
Deployment and Tool Comparison
Ollama serves as a headless server that can also be containerized via Docker for service-oriented deployments. While Ollama is optimized for local CLI workflows, it is distinct from vLLM, which is better suited for high-throughput production serving due to advanced memory management and request batching. For users preferring a graphical interface, LM Studio offers a similar local model experience, though Ollama remains the standard for CLI-integrated agent workflows.