From Gradients to ChatGPT¶

Cover illustration: a circus-themed map of the course. Numbered booths under a big top represent each of the twenty modules — pretraining, tokenization, embeddings, gradient descent, self-attention, multi-head attention, the transformer, sampling, SFT, DPO, RAG, tools, the agent, the eval inspector, and an "inference booth" robot at the bottom.

A self-study course building a tiny LLM stack from scalar autodiff up through a working chat assistant, in two parts. Modeled after From NAND to Tetris: the codebase grows layer-by-layer, and every block of the stack is something you write yourself.

The hard constraint: everything runs on an M-series MacBook — no cloud GPUs, no paid compute.

Why this course¶

Millions interact with LLMs on a daily basis. But few bother to understand how these systems actually work. How does an LLM understand language? What is a "model" and how does it learn? How does a chat assistant "know" how to answer a question? How does matrix multiplication produce intelligent behavior? In this course we answer these questions from first principles.

Our goal is to go below the API. To understand every part of the LLM stack, end to end. You start with basic autodiff, and over twenty modules grow it: tensors, a neural net, a tokenizer, embeddings, attention, the transformer, pretraining, sampling, SFT, DPO, evaluation, RAG, tool use, an agent loop. By the final module you have a chat assistant you built end to end — tools, retrieval, and a swappable backend: the tiny model you trained yourself, or a stronger local open model when you want it to be genuinely useful. All on your laptop. All built by you.

The scaffolding is real but small: a tokenizer that takes ten minutes to train, a transformer with a few million parameters, a corpus that fits in RAM. Tiny is deliberate. Once you've built every layer once at toy scale, the production-scale versions stop being magic.

What's in it¶

Twenty modules plus a fast prerequisite review, in two parts. Each module is roughly one week of effort at the level of a rigorous elite-college course.

Part I — From gradients to a language model¶

Build the model itself. You end with a language model whose every layer you wrote, generating readable text on your laptop.

#	Module	Group
00	Prerequisite review	Prerequisite review
01	Scalar autodiff	Foundations
02	Tensors and matmul	Foundations
03	A first neural network	Foundations
03B	Training	Foundations
04	Tokenization	Language
05	Embeddings and positions	Language
06	Next-token prediction	Language
07	Self-attention	The transformer
08	Multi-head attention	The transformer
09	The transformer block	The transformer
09B	Pretraining	The transformer
10	Milestone: TinyLLM	The transformer
11	Sampling and decoding	The transformer

Part I ends here, and finishing it is a real accomplishment. You will have built a working language model from scalar derivatives up, with no black boxes in the path.

Part II — From a language model to ChatGPT¶

Build the system around a model. A different subject from Part I, not a harder one: less derivation, more systems engineering.

#	Module	Group
12	Scaling experiments	Behavior shaping
13	Instruction tuning (SFT)	Behavior shaping
13B	LoRA	Behavior shaping
14	Preference tuning (DPO)	Behavior shaping
15	Hallucination and evaluation	Behavior shaping
16	Local pretrained models and inference	Assistant systems
16B	Synthetic data	Assistant systems
17	Retrieval-augmented generation	Assistant systems
18	Tool use	Assistant systems
19	Agent loops	Assistant systems
20	Capstone: a tiny ChatGPT	Assistant systems

Part II can also be entered directly if you already know the fundamentals and want the systems material — see Module 12.

The syllabus lays out each module in detail and gives the full motivation for the ordering.

Who it's for¶

You'll get the most out of this if you're comfortable with Python, undergraduate calculus (chain rule, gradients), and basic linear algebra. You don't need prior deep learning experience — Module 0 covers the prerequisites and Modules 1–3 build the math substrate from scratch. You should be willing to read a paper now and then, and be willing to debug your own code without a framework hiding the failure mode.

If you've watched Karpathy's videos and wished for a structured curriculum with exercises, deliverables, and tests, this is that.

How the course works¶

Each module in the course has a lesson page and a set of deliverables combining a coding project and a problem set. The lesson pages are hosted on this site and also available as markdown in the course repo.

Each week, students will read the module lesson page. They'll then implement a new sub-package covering that week's topic inside the g2c/ Python package. Finally they'll complete a set of student exercises in a Jupyter notebook using the code they wrote that week. The lesson pages are the readable front door; the repo is where the code lives.

Get started¶

Read the syllabus for the full arc across both parts.
Clone the repository and follow the README quickstart to set up on your machine.
Start with Module 0: Prerequisite review, or jump straight into Module 1: Scalar autodiff.

If you get stuck¶

Two things, in this order.

Keep moving. You don't have to debug backwards through five modules to make progress. Hand back the reference implementations for the modules you aren't working on and carry on with your own code everywhere else:

./notebook.sh 12 --solutions=01-07

Then tell us where it happened. Open a stuck report naming the module. Nothing in the course phones home, so a module that quietly loses people is invisible to us unless someone says so. It genuinely helps, and it takes a minute.