Tiny LLMs on microcontrollers
Language models running on ESP32 microcontrollers, measured honestly, and a two-board story box that listens, tells a story, and speaks it aloud.
How small can a useful language model get? This project starts from slvDev’s esp32-ai, a 28.9-million-parameter model running on an ESP32-S3, and pushes it onto Espressif’s faster ESP32-P4.
Benchmarks first
Before anything clever, I wanted real numbers: tokens per second on the S3 and the P4, and the energy each token costs, measured at the USB port. The first write-up is on InsiderLLM. I’ve opened an issue upstream to contribute improvements back, and the P4 port will get its own repository.
The P4 firmware has moved from Arduino to native ESP-IDF, to get at all 32 MB of flash.
The story box
A model that small won’t answer your email, but it can tell a story. The TinyStories research showed that very small models can write coherent children’s stories, so I’m extending that idea so the model answers a plain request: “tell me a story about a fox.”
The plan is two boards:
- The storyteller: one P4 running the story model, trained on one of my GPU machines at home.
- The voice: a second P4 for speech recognition and speech synthesis. After listening tests, TinyTTS at 44.1 kHz is the voice to beat. A custom voice distilled from Kokoro comes next, and maybe a real voice actress much later.
Bringing the two together is a long way off. For now each board has its own bench.
What’s next
Beyond the P4, the STM32N6, with its built-in neural accelerator, looks like the next board to try. I’m still looking for the killer app for microcontroller LLMs; voice commands that never leave the device are a strong candidate.