Workshop on Building a Language Model from Scratch Using nanoGPT

The workshop guides participants through writing a complete GPT training pipeline, emphasizing hands-on experience with each component.

Using the nanoGPT framework, the workshop aims to reproduce a smaller version of GPT-2 with approximately 10 million parameters, which can be trained on a laptop in less than an hour.

Training can utilize Apple Silicon GPUs, NVIDIA GPUs, or CPUs, and participants can also use Google Colab for training.

The workshop covers character-level tokenization, which is suitable for smaller datasets like Shakespeare's texts, and introduces the option to switch to BPE tokenization for larger datasets in later sections.

Participants will create essential scripts including model.py, train.py, and generate.py, gaining a comprehensive understanding of the training process.