Running a 28.9M Parameter LLM on an $8 ESP32-S3

Memory Optimization via Per-Layer Embeddings

The primary constraint for running large language models on microcontrollers is the limited SRAM. The ESP32-S3 provides only 512KB of SRAM, which is insufficient for storing a 28.9-million-parameter model. The solution uses a per-layer embedding technique, where the large embedding table (comprising the majority of the model's parameters) is stored in the chip's 16MB of flash memory rather than in SRAM. During inference, the system only loads the specific rows needed for the current token into SRAM, while the attention heads and feed-forward layers remain in the fast memory for computation. This approach allows the model to perform inference at approximately 9 tokens per second.

Implementation and Training

The project relies on the TinyStories dataset, which is specifically curated for small models to produce coherent narratives. The inference engine is built on the principles of the llama2.c project, using portable C code to run without an operating system or Python interpreter. To replicate this, one must use the ESP32-S3-N16R8 variant, which features 16MB of flash and 8MB of PSRAM. The training process involves exporting the model weights and flashing them directly to the device. Because the model is pre-flashed with specific weights, changing the input prompt requires reflashing the entire firmware, and the model's output is heavily biased toward the training data, often reverting to the same story regardless of the prompt provided.