08 - Training LLMs from Scratch

This module puts the earlier concepts into practice by building and training language models end to end. It opens with a primer that follows the whole of Andrej Karpathy's Let's build GPT lecture alongside an interactive visualizer, from a bigram lookup table to a full Transformer, with thanks to him for the lecture and its MIT-licensed code. Each chapter after that is a complete build of one model version, taking it from raw text to a trained model that generates text. It starts with a GPT-style decoder, and later chapters add newer architectures and larger training recipes on top of the same pipeline.

Learning objectives 3-4 hours (growing as builds are added)

By the end of this module you will be able to:

Trace a minimal language model (a bigram lookup table) from (B, T) token ids to logits, loss and sampled text
Run every step of a from-scratch pretraining pipeline in the right order, from data cleaning to sampling
Split train and validation data without leakage, and build (x, y) context blocks and batches from a token stream
Implement and size a GPT-style decoder, predicting its parameter count and training compute before running it
Train it with a correct loop (AdamW, LR schedule, gradient accumulation, clipping, bf16) and diagnose it from the loss curves

Prerequisites

Foundation Models - tokenization, the transformer and attention
PyTorch Fundamentals - the training loop and the GPT From Scratch lab
Pretraining at Scale - objectives, data curation and scaling laws

Chapter Map

#	Note	Topic	Level
1	Primer: Let's Build GPT with Andrej Karpathy, Visualized	All of Karpathy's Let's build GPT (0:00-1:56:20) in four parts, with a synced full-screen lab: a bigram baseline (tokenizer, (B, T) batches, the 65 × 65 table, loss ≈ ln 65 at init), self-attention derived step by step, the Transformer block (residuals, LayerNorm, dropout, scaling up) and how pretraining plus RLHF becomes ChatGPT	Beginner
2	GPT From Scratch: Step by Step	The full pipeline: plan the run (C ≈ 6ND), clean and dedup, tokenize, split train/val before chunking, build (x, y) blocks and batches, a GPT decoder in PyTorch, sanity checks, AdamW + cosine, the training loop, evaluation, checkpointing and sampling	Intermediate

Planned builds: later chapters will follow the same step-by-step format for other model versions, such as a full GPT-2 (124M) reproduction on FineWeb-Edu, a modern Llama-style decoder, a mixture-of-experts model and training a tokenizer from scratch.

Code Lab

The runnable companion for this module is the GPT From Scratch lab in PyTorch Fundamentals. It is a modernized version of the model built in chapter 2, with RMSNorm, RoPE, grouped-query attention and SwiGLU, trained on Tiny Shakespeare, and a smoke test that runs on a laptop CPU.

Summary & Key Terms

Summary & Key Terms - a quick recap of this module and its essential vocabulary.

Next Topic

Previous: 07 - Pretraining at Scale · Next: 09 - Post-Training & Fine-Tuning