Fine-Tuning Tiny LLMs for On-Device Agents

AI Engineer2026-05-20go watch the original →

the gist

Developers can achieve 90% accuracy on function-calling tasks by fine-tuning 270M parameter models on synthetic datasets, significantly outperforming out-of-the-box prompting.

Fine-Tuning for Task-Specific Accuracy

For boutique or highly specific tasks, developers should move beyond generic system prompts and fine-tune small language models (SLMs) under 1 billion parameters. Using Function Gemma (270M parameters) as a base, the author demonstrates that out-of-the-box performance on a fixed set of app intents yields approximately 46% accuracy. By generating a synthetic dataset to train the model specifically on those intents, accuracy increases to over 90% for eight out of ten functions, with the remaining two reaching the 80% range.

Implementation Workflow

Model Selection: Utilize the LiteRT-LM runtime to package and ship models directly within an application, supporting execution on CPU, GPU, or NPU.
Synthetic Data Generation: Create a focused dataset using tools like Flash to simulate the specific function-calling requirements of the application.
Fine-Tuning: Use the Function Gemma fine-tuning lab (available as a Hugging Face space) to train the model on the synthetic data, ensuring robust performance even on legacy hardware like the Pixel 7.
Agent Harnessing: Implement an agent harness that loads skill descriptions on demand. The model uses a tool-calling mechanism to select the appropriate skill based on user input, which then triggers custom JavaScript or native code to execute the task.

On-Device Runtime Options

System-Level GenAI: Leverage Gemini Nano via AI Core for common tasks. This approach is pre-installed, highly optimized, and does not increase the application binary size.
Custom App-GenAI: Use LiteRT-LM for custom, boutique models that require full control and specialized fine-tuning. This runtime supports cross-platform deployment, including Android and iOS.