https://zheng-bobo.github.io/en/tags/llm-training/ Recent content in LLM Training on Blowing in the wind Hugo -- gohugo.io en Sat, 26 Sep 2026 21:00:00 +0200 https://zheng-bobo.github.io/en/post/nanochat-gpt-pretraining-from-source/ Sat, 26 Sep 2026 21:00:00 +0200 https://zheng-bobo.github.io/en/post/nanochat-gpt-pretraining-from-source/ <p>This article follows the execution order of <code>scripts/base_train.py</code>: command-line arguments, random seeds, DDP setup, model construction, weight initialization, scaling laws, optimizers, data loading, gradient accumulation, and finally one complete training step.</p> <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre class="chroma"><code class="language-text" data-lang="text"><span class="lnt">1 </span><span class="lnt">2 </span><span class="lnt">3 </span><span class="lnt">4 </span></code></pre></td> <td class="lntd"> <pre class="chroma"><code class="language-text" data-lang="text">documents → tokenizer and sequence packing → inputs/targets → GPT forward → cross-entropy loss → backward and gradient accumulation → optimizer step → evaluation, sampling, checkpointing, and resume</code></pre></td></tr></table> </div> </div> <p>The organization and code-reading path follow my nanochat Notion notes. The prose has been edited for clarity, while the source snippets and their original reading sequence are preserved.</p> <blockquote> <p><strong>Prerequisite:</strong> If token embeddings, causal self-attention, MLPs, residual connections, or next-token loss are still unfamiliar, start with <a href="https://zheng-bobo.github.io/en/post/transformer-architecture/">Transformer Architecture: From Token Embedding to the Training Loop</a> and then return to this source-level walkthrough.</p> </blockquote>