Running Local LLMs on Apple Watch Series 6
Better Stackgo watch the original →
the gist
By bypassing Core ML and compiling llama.cpp for the arm64_32 architecture, you can run small state-space models like Falcon H1 locally on an Apple Watch with performance reaching 24 tokens per second.
Bypassing Core ML for State-Space Models
Standard Apple development relies on Core ML, but it fails for models like Falcon H1 because it lacks support for state-space layers (Mamba-2). Core ML requires a fixed graph decided at compile time, whereas the Mamba-2 scan loop depends on conversation length, making it incompatible with Apple's static conversion process. While attention layers are easily converted, the hybrid architecture of Falcon H1 forces a manual approach to inference.
Compiling llama.cpp for watchOS
Although llama.cpp does not officially list watchOS as a supported target, the arm64_32 architecture is fully capable of running the required math operations. The primary constraint is the 32-bit memory addressing limit, which caps total addressable memory at 2 GB. To deploy, one must compile llama.cpp using specific build flags for the arm64_32 target. The resulting binary can execute quantized GGUF models like the 90M parameter Falcon H1 or the 135M parameter SmolLM2 directly on the watch hardware.
Performance and Tooling
On an Apple Watch Series 6, the Falcon H1 model achieves approximately 24 tokens per second for simple queries. The implementation supports tool calling by registering a custom tools file, allowing the model to trigger functions like Wikipedia lookups or weather reporting. Performance remains stable for short interactions but degrades as context length increases, with token generation speeds dropping toward 10 tokens per second during complex, multi-paragraph explanations.