Model reports often list ARC-E, ARC-C, MMLU, GPQA, GSM8K, HumanEval, SWE-bench, GAIA, WebArena, and OSWorld side by side. Their scores are not interchangeable: each benchmark uses different tasks, tools, environments, inference budgets, and scoring rules.

This guide maps common evaluations from static question answering to agents completing real tasks, and explains what each benchmark can—and cannot—tell us.

This article follows the execution order of scripts/base_train.py: command-line arguments, random seeds, DDP setup, model construction, weight initialization, scaling laws, optimizers, data loading, gradient accumulation, and finally one complete training step.

1
2
3
4

documents → tokenizer and sequence packing → inputs/targets
          → GPT forward → cross-entropy loss
          → backward and gradient accumulation → optimizer step
          → evaluation, sampling, checkpointing, and resume

The organization and code-reading path follow my nanochat Notion notes. The prose has been edited for clarity, while the source snippets and their original reading sequence are preserved.

Prerequisite: If token embeddings, causal self-attention, MLPs, residual connections, or next-token loss are still unfamiliar, start with Transformer Architecture: From Token Embedding to the Training Loop and then return to this source-level walkthrough.

Transformer appears to be made up of many components, but a GPT-style forward path can be summarized as:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

Token IDs
   ↓
Token Embedding + Position Embedding
   ↓
[LayerNorm → Self-Attention → Residual]
   ↓
[LayerNorm → FFN → Residual]
   ↓
Repeat for multiple Transformer blocks
   ↓
Final LayerNorm → LM Head
   ↓
Next-token logits → Cross-Entropy Loss

This article starts from a simple GPT implementation and explains the role and tensor shape of each step along the data flow. The complete source code is placed at the end of the article. After reading the previous principles, you can compare and understand it with the code.

“Agent” is often used as a broad label for any system that calls tools or takes multiple steps. Anthropic’s engineering work offers a more useful distinction: a workflow follows code paths defined in advance; an agent lets the model dynamically choose its process from environmental feedback.

This first article in the “Anthropic Agent Evolution” series establishes the vocabulary: augmented LLMs, fixed workflows, autonomous agents, and the harness that surrounds the model.

The central tension in long-running work is not whether a model can write code. It is that a task may last hours or days while every context window is finite. Replaying the entire history eventually creates both capacity and noise problems.

Anthropic’s first long-running agent harness answered with a simple principle: reset conversational context, but externalize project state into structured artifacts that the next agent can recover.

The first harness let an agent continue across context windows, but continuity is not quality. Generators routinely mistake “mostly runs” for “good enough,” especially in subjective product and design work.

Anthropic’s next step separated planning, generation, and evaluation so that an external evaluator became a source of evidence the generator had to respond to.

Earlier harnesses focused on how a model could continue work and improve quality. A managed service introduces system questions: how is a session persisted, how does work recover after a harness crash, how can sandboxes be replaced, how do customer VPCs connect, and how are credentials kept away from model-generated code?

Anthropic Managed Agents answers by decoupling the brain, hands, and session behind stable interfaces.

Self-Improving AI Agent is not a mysterious system that recursively modifies itself indefinitely. A more practical understanding is: Agent forms a closed loop between generation, action, observation and verification, and uses the feedback obtained during reasoning to improve current answers, subsequent decisions, and even the next round of training.

This article follows the main line of the Stanford CS329A course and strings together test-time calculations, verifiers, tool feedback, planning search, reinforcement learning, in-depth research and long-term evaluation into a complete framework.