nanochat's GPT-2 Speedrun Is Down to 1.65 Hours on 8 H100s

Andrej Karpathy’s nanochat repo now trains a GPT-2-grade model in 1.65 hours on a single 8xH100 node. That is the top of its “time-to-GPT-2” leaderboard, a run dated March 14, 2026, tagged “autoresearch round 2” and credited to Karpathy. The baseline it beats is OpenAI’s original 2019 GPT-2 checkpoint, which scored 0.2565 on the DCLM CORE metric and, per the repo, cost roughly $43,000 to train. At the listed ~$3/GPU/hr, an 8xH100 node runs about $24/hr, so two hours is roughly $48. On a spot instance, nanochat’s own README puts the total closer to $15.

The number that matters is not the dollar figure. It is the shape of the leaderboard. Six entries, five of them Karpathy’s, and the time falls from 3.04 hours on Jan 29 to 1.65 hours on Mar 14. The CORE scores barely move: 0.2585, 0.2578, 0.2602, 0.2571, 0.2690, 0.2626. This is not a story about a better model. It is a story about a cheaper, faster way to reach a fixed capability bar, and about a public scoreboard that turns that pursuit into a game a single person can play.

One dial, and everything else follows

The design choice that makes nanochat interesting is its refusal to expose knobs. You set --depth, the number of transformer layers, and the harness computes width, head count, learning-rate schedule, training horizon, and weight decay automatically so the model comes out compute-optimal. GPT-2 capability lands around depth 24 to 26. The repo’s framing is that the user is “simply asking for a smaller or bigger model” and everything “just works.”

That is a real editorial position, not a convenience feature. Most training codebases are graveyards of hyperparameters that only their authors understand. nanochat says the interesting variable is scale, and once you fix scale, the rest is a solved function. If that holds, it lowers the floor for anyone who wants to run a serious training experiment: a researcher who wants to test an idea at GPT-1 scale can run a 12-layer model in about five minutes of pretraining, per the README’s suggested loop, and watch val_bpb, core_metric, and MFU in wandb.

The catch is baked into the same design. Any change to the repo “has to be principled enough that they work for all settings of depth.” You cannot tune a trick for one size and ship it. That constraint is what makes the leaderboard honest, and it is also what will slow contributors down.

The leaderboard is the actual product

Look at what moved the time down. Entry 4, dated March 4, swaps the dataset to NVIDIA ClimbMix and cuts the run from 2.76 to 2.02 hours. That is a bigger jump than any architecture tweak on the board. Entries 5 and 6, the “autoresearch” rounds, shave it to 1.80 and then 1.65. The board does not tell us what autoresearch changed, which is a gap worth naming. A leaderboard that reports time and CORE but not the mechanism of each win is a scoreboard, not a lab notebook.

Still, the incentive structure is the point. Karpathy cites the modded-nanogpt repo as the inspiration, and the pattern is the same: publish a reference run, publish a clock, let the community race it. The runs/speedrun.sh script is the canonical entry, and the board is seeded with the original OpenAI checkpoint at 168 hours so every entry has a fixed target to beat. That is a cleaner experimental setup than most academic benchmarks, because the metric (wall-clock time to a fixed CORE threshold) is hard to game and easy to reproduce.

What $48 actually buys, and what it doesn’t

The README is refreshingly blunt about the output. The speedrun model is “a 4e19 FLOPs capability model so it’s a bit like talking to a kindergartener.” The sample conversation shows it explaining Rayleigh scattering as “an optical illusion” made of “tiny blue and violet particles,” then writing a poem that rhymes “blue” with “blue” three times. This is GPT-2. It is not a product. Anyone reading the “$100 ChatGPT” tagline as a claim about a usable assistant is misreading it, and the repo does not encourage that misreading.

What the $48 does buy is a full pipeline in one place: BPE tokenizer training, pretraining, SFT, RL, evaluation, and a KV-cache inference engine, all in a file tree small enough to read in an afternoon. The tasks/ directory carries ARC, GSM8K, MMLU, HumanEval, and SmolTalk. The nanochat/optim.py file implements AdamW plus Muon, single-GPU and distributed. Precision is handled explicitly through a single COMPUTE_DTYPE global rather than torch.amp.autocast, with bf16 on SM 80+ hardware and fp32 fallbacks on older cards. These are the choices of someone who has debugged training runs at scale and wants the next person not to repeat it.

The leaderboard measures time to a fixed capability bar. It does not measure whether the bar is the right one.

The cost curve is the real story

Seven years from $43,000 to under $100 for the same capability is the headline, and it is not only about cheaper GPUs. It is dataset curation (ClimbMix), precision choices (fp8 in entry 2), batch-size tuning (entry 3), and a lot of small optimizations that only show up when someone is timing the whole run end to end. The repo’s own framing, “many advances over 7 years across the stack,” is the correct one. No single trick did this.

The uncomfortable implication for the AI economy is that the capability floor keeps dropping while the frontier keeps rising. A GPT-2-grade model is now a weekend project and a rounding error. That does not threaten OpenAI or Anthropic, whose business is at the top of the curve. It does threaten the assumption that training a competent small model requires a lab. It also sharpens a policy question: if a $15 spot-instance run produces a model that can write plausible text, the compute-threshold regimes some regulators have floated are measuring the wrong thing. Capability, not cost, is the variable that matters, and nanochat shows how fast the two decouple.

What to watch

The leaderboard’s next entries will tell us whether the 1.65-hour mark is a plateau or a waypoint. The gap between entry 1 (3.04 hours) and entry 6 (1.65 hours) is roughly a 1.8x speedup in about six weeks, which is a pace that cannot continue indefinitely but has not visibly slowed. Watch whether outside contributors start landing entries, and whether the repo begins reporting what each win changed. A speedrun leaderboard without mechanism notes is a spectator sport. With them, it becomes a public record of how LLM training actually gets faster, one commit at a time.