MLX Powers On-Device AI on Apple Silicon

AI Engineer2026-05-11go watch the original →

the gist

MLX runs real-time vision, <100ms TTS, speech-to-speech, omni models like Gemma 4E, and 1M-context LLMs on Macs/iPhones via Turbo Quant's 4x KV cache cut; demos modular pipelines and community robotics.

The Breakthrough

Prince Canuma demonstrates MLX, an array framework for Apple Silicon that runs frontier open-source models like Gemma 4 entirely on-device, with Turbo Quant reducing KV cache by 4x to enable 1M context lengths on local hardware.

What Actually Worked

MLX VLM performs real-time vision analysis; a simple Python command runs the RFD model by Rooflow to detect and describe objects like glasses or masks offline on Mac or iPhone.
Modular speech-to-speech pipelines chain user-selected ASR, LLM, and TTS models; supports Python for rapid prototyping and Swift for native apps, adjustable to hardware like M1 Macs.
Marvvis TTS generates audio in under 100ms; combines with real-time STT (ports of Whisperflow or Super Whisper via mlx-audio) for voice-controlled computing.
Omni models process image, audio, and text inputs; Gemma 4E variants and Qwen 3 Omni (30B parameters) run on-device, including Gemma 4 26B on iPhone storage for reasonable speeds.
mlxvlm ui launches Gradio interface for Gemma 4; analyzes uploaded images offline using GPU, describing details like profile photos and bios.

Before / After

Turbo Quant shrinks KV cache from nearly 1GB (full model) to 1/4 size with exact match performance; at 300,000 context, throughput nearly doubles.

Context

Prince Canuma built on-device AI to restore accessibility for his blind father in Africa, where cloud services fail due to unreliable internet and poor subscriptions. Apple Silicon's chips enable this local compute, unlike cloud-optimized frameworks from Meta or Google. MLX has 1.5M downloads, 4,000+ ported models, and day-zero support for releases like Gemma 4. Demos prove viability for agents, voice apps, security cams, and robots without cloud dependency. Community projects extend to chained video generation on 16GB VRAM Macs, grounded visual reasoning for dash cams, native voice apps like Locally, and real-time voice-cloned robots like Reachy Mini powered by MLX audio/vision.

Performance Monitoring and Limits

Mactop overlays show GPU/CPU usage during inference (e.g., GPU spikes on 'hi' prompts). MLX prioritizes GPU over Neural Engine (CoreML lacks easy dev experience); hybrid planned post-WWDC. Open-source models trail Cloud 3 Opus but close gaps quickly; adjust expectations for use cases like parallel image inference or long docs.

Notable Quotes

"All you need to pay is your energy bill."
"Turbo Quant... reduce KV cache by 4x... at 300,000 context the performance almost doubles in terms of throughput."
"You can run models of hundreds of billions of parameters even on your initial M1 MacBook."

Content References

Wait, no: this is in JSON, but integrated? No, separate field.