Deploying Tiny LLMs and Agents on Edge Hardware

The DRAM Constraint

Edge AI development is primarily limited by DRAM availability and cost rather than raw compute power. As mobile manufacturers reduce memory capacity and hardware costs for devices like the Raspberry Pi rise, developers must prioritize model footprint reduction. The Google AI Edge team addresses this by using aggressive quantization and selecting the smallest possible model for specific tasks to ensure compatibility with devices that have less than 2GB of RAM.

Small vs. Tiny Model Playbooks

For models in the 1 to 4 billion parameter range, the standard approach is zero-shot prompting or using LoRA adapters. These models are capable of reasoning and function calling but typically require 4GB+ of DRAM. When these models are too large for the target hardware, the team shifts to "tiny" models (50M to 500M parameters). Unlike larger models, these require fine-tuning to achieve high reliability. The team uses synthetic data generation to create datasets ranging from 10,000 to 10 million samples to specialize these models for specific tasks like function calling or text cleanup.

Real-World Edge Applications

Fine-tuned tiny models enable offline features that previously required cloud subscriptions. For example, a voice-to-function calling agent can achieve over 86% reliability by mapping free-text input to a set of 10 predefined functions. Similarly, offline voice dictation apps can be built using two sub-billion parameter Gemma models that handle both ASR and text post-processing to remove filler words. These models can achieve decode speeds of approximately 45 tokens per second on edge devices, significantly outperforming the single-digit speeds seen with larger 2B parameter models.