Android On-Device AI: Gemini Nano and Hybrid Inference

Centralized Model Management via AI Core

Google handles the deployment and execution of Gemini Nano through the AI Core system service. By centralizing the model at the OS level, the system avoids the need for individual apps to bundle large model weights, which typically range from 3GB to 4GB. This architecture allows the system to manage resource contention, prioritize foreground app requests, and queue background batch tasks to run during charging cycles. Developers interact with these capabilities through the ML Kit GenAI APIs, which abstract away the underlying hardware acceleration and TPU configuration.

Hybrid Inference and Model Reach

To address the hardware requirements of Gemini Nano, which is currently limited to flagship devices, Google introduced a hybrid inference model via Firebase AI. This approach allows developers to implement a fallback mechanism: if the on-device Gemini Nano model is unavailable on a specific device, the system automatically routes the inference request to Gemini Flash in the cloud. While classical ML Kit models for vision and OCR support over a billion devices, GenAI features currently require recent flagship hardware. Developers needing broader reach or custom model support can utilize LiteRT, though this requires manual testing and optimization across the device matrix.

Future API Extensions

Google is actively expanding the on-device toolset to support RAG-style architectures. While the current Prompt API supports text and image inputs, an embedding API is in development to enable vectorization and similarity search directly on the device. This will allow developers to build more complex, context-aware applications without offloading sensitive user data to cloud servers.