Reading nanochat Source: From Configuration to a Training Step

Sat, 26 Sep 2026 21:00:00 +0200

This article follows the execution order of scripts/base_train.py: command-line arguments, random seeds, DDP setup, model construction, weight initialization, scaling laws, optimizers, data loading, gradient accumulation, and finally one complete training step.

documents → tokenizer and sequence packing → inputs/targets
          → GPT forward → cross-entropy loss
          → backward and gradient accumulation → optimizer step
          → evaluation, sampling, checkpointing, and resume

The organization and code-reading path follow my nanochat Notion notes. The prose has been edited for clarity, while the source snippets and their original reading sequence are preserved.

Prerequisite: If token embeddings, causal self-attention, MLPs, residual connections, or next-token loss are still unfamiliar, start with Transformer Architecture: From Token Embedding to the Training Loop and then return to this source-level walkthrough.

Transformer Architecture: From Token Embeddings to the Training Loop

Mon, 14 Sep 2026 10:25:02 +0200

Transformer appears to be made up of many components, but a GPT-style forward path can be summarized as:

Token IDs
   ↓
Token Embedding + Position Embedding
   ↓
[LayerNorm → Self-Attention → Residual]
   ↓
[LayerNorm → FFN → Residual]
   ↓
Repeat for multiple Transformer blocks
   ↓
Final LayerNorm → LM Head
   ↓
Next-token logits → Cross-Entropy Loss

This article starts from a simple GPT implementation and explains the role and tensor shape of each step along the data flow. The complete source code is placed at the end of the article. After reading the previous principles, you can compare and understand it with the code.

GPT on Blowing in the wind

Reading nanochat Source: From Configuration to a Training Step

Transformer Architecture: From Token Embeddings to the Training Loop