New language model runs on $8 microcontroller with 28.9 million parameters

The language model generates text at a rate of about 9 tokens per second and displays it on a small screen connected to the ESP32-S3 microcontroller.

Previous models on similar chips were limited to around 260,000 parameters due to memory constraints, but this new model leverages Google's Per-Layer Embeddings to store most parameters in flash memory, allowing for a larger model size.

The ESP32-S3 has 512KB of SRAM, which is used for the model's core processing, while a 25 million row embedding table resides in slower flash memory.

The model was trained on a dataset called TinyStories, enabling it to write short, coherent stories, but it is not capable of answering questions or performing complex tasks due to limitations in its reasoning capabilities.

The implementation details, including firmware and training methods, are documented in a repository, which also includes a history of development and corrections made during the process.