Image by Nana Dua on Unsplash Source

Updated October 23, 2026: added the repo’s status seven months after launch (about 98,000 stars, no commits since March 26), autoresearch round 2 on nanochat, Shopify’s pi-autoresearch, Karpathy’s AgentHub sketch, the research papers that now use AutoResearch as a baseline, the main criticisms, and corrected several details of the original results.

Karpathy’s AutoResearch is a small open-source repo that lets an AI coding agent do machine-learning research unattended. The agent edits one 630-line training script, trains a small GPT for exactly five minutes on one GPU, keeps the change if validation bits per byte (val_bpb) went down, reverts it if not, and repeats: about 12 experiments an hour, roughly 100 overnight. Andrej Karpathy announced it on March 7, 2026. Seven months later the repo has about 98,000 GitHub stars and 13,600 forks but no commits since March 26. Karpathy left it as a seed, and the action moved to forks, to Shopify’s general-purpose version, and to a stack of research papers that now treat AutoResearch as the baseline to beat.

The headline result holds up. In roughly 700 autonomous experiments over two days, Claude found about 20 changes to Karpathy’s already hand-tuned nanochat that stacked and transferred to bigger models, cutting nanochat’s “time to GPT-2” from 2.02 to 1.80 hours. A second round five days later took it to 1.65 hours. What has changed since March is the verdict on the method: the loop works, but it is a greedy hill-climber that gets stuck, can be gamed, and mistakes noise for progress unless you design around those failure modes.

Related: Agentic AI in R&D: How Agent-Accelerated Research Changes Competitive Advantage

How AutoResearch Works: Three Files, One Metric, Zero Handholding

The design philosophy behind AutoResearch is radical simplicity. The whole system consists of three files that matter:

train.py (630 lines): Contains the GPT model definition, the Muon+AdamW optimizer, and the training loop. This is the only file the agent is allowed to edit. Every experiment is a modification to this file.

prepare.py: Handles data preparation, tokenization, and defines the evaluate_bpb() function. The agent is told it may read this file but not modify it, so that it cannot move the goalposts on the evaluation metric.

program.md: Natural language instructions that tell the agent what to do. Karpathy describes it as “essentially a super lightweight ‘skill’”: the human iterates on this file, the agent iterates on train.py. It includes one critical directive: “Do NOT pause to ask the human if you should continue. You are autonomous. The loop runs until the human interrupts you, period.”

The single metric is val_bpb (validation bits per byte). Lower is better. Because it is vocabulary-size-independent, the agent can make architectural changes (adding or removing layers, changing attention mechanisms) and still get a fair comparison across experiments.

The Eight-Step Loop

Each experiment follows an identical cycle, spelled out in program.md:

  1. Analyze: Read train.py, check git state, review previous experiment results
  2. Hypothesize: Propose a specific modification (architecture tweak, hyperparameter change, optimizer adjustment)
  3. Modify: Edit train.py with the proposed change
  4. Commit: Git commit the modification
  5. Train: Execute uv run train.py > run.log 2>&1 with a fixed 5-minute training budget (runs over 10 minutes get killed)
  6. Extract: Grep val_bpb and peak memory from the training log; if the grep is empty, read the stack trace and try to fix the crash
  7. Evaluate: Compare against the previous best score
  8. Decide: If improved, keep the commit and advance the branch. If equal, worse, or crashed, reset to where it started. Return to step 1.

Every result gets logged to an untracked results.tsv with a status of keep, discard, or crash. This gives the agent a growing history of what has been tried and what worked, which informs future hypotheses.

One rule in program.md gets less attention than it deserves: a simplicity criterion. “A 0.001 val_bpb improvement that adds 20 lines of hacky code? Probably not worth it. A 0.001 val_bpb improvement from deleting code? Definitely keep.” The agent is asked to weigh complexity against gain, which is exactly the judgment call that separates a useful research log from a pile of overfit hacks.

Why 630 Lines Matters

The size is a deliberate choice. All of train.py fits comfortably in a current model’s context window, so the agent reasons about the whole training setup at once instead of guessing at interactions between files it hasn’t read. Karpathy’s own justification in the README is more modest: one editable file “keeps the scope manageable and diffs reviewable.”

The fixed time budget does the rest. Because every run gets exactly five minutes of wall-clock training, a bigger model and a smaller-but-faster one are compared on equal terms, and the agent ends up optimizing for whatever is best on your GPU in that window. The flip side, which the README states openly: your results are not comparable to anyone else’s on different hardware.

The rules against touching prepare.py, installing packages, or changing the evaluation are what make the results credible. They are also just prose in a Markdown file, which turned into one of the main criticisms (more on that below).

What AutoResearch Actually Found

Karpathy did not release AutoResearch as a toy demo. He ran it against nanochat, his “best ChatGPT that $100 can buy” project, which keeps a “time to GPT-2” leaderboard: the wall-clock time an 8xH100 node needs to beat GPT-2’s CORE score. The first four entries were Karpathy’s own manual tuning work (one with a community contributor). Entry five, dated March 9, is “autoresearch round 1,” and it beat all of them.

Over ~700 experiments across two days on depth-12 models, the agent found about 20 improvements that stacked. Karpathy’s summary and the commit itself list them:

  • QKNorm was missing a scaler multiplier, so attention was too diffuse. The agent added post-QK-norm scaling (q and k multiplied by 1.15) to sharpen it.
  • Value embeddings had no regularization at all. “Oops,” in Karpathy’s words.
  • The banded attention window had never been tuned. The commit shrinks the short attention window from half to about a third of the sequence length.
  • AdamW betas were “all messed up.” The fix was per-group betas and weight decay instead of shared global values.
  • Weight decay schedule, RoPE base and initialization got retuned: cosine instead of linear weight decay, RoPE theta from 10K to 100K, smaller embedding and MLP init.

Together they brought “time to GPT-2” from 2.02 hours down to 1.80 hours, about 11%. All of them transferred from depth-12 to depth-24 models without re-tuning. Karpathy’s own framing is worth quoting because it is more careful than most of the coverage: “It’s not novel, ground-breaking ‘research’ (yet), but all the adjustments are ‘real’, I didn’t find them manually previously, and they stack up and actually improved nanochat.”

Round 2 followed on March 14. The commit added two small architectural ideas, “smear” (mixing the previous token’s embedding into the current position through a learned gate, a cheap bigram signal) and “backout” (subtracting a learned fraction of a mid-layer residual before the output projection), plus more hyperparameter tuning. Time to GPT-2 dropped to 1.65 hours, 99 minutes. As of October 2026 that is still the top entry on the nanochat leaderboard.

Related: Software Factories: When AI Agents Build Software Without Human Review

Beyond Karpathy: Tobi Lutke’s Results

Shopify CEO Tobi Lutke applied the pattern to two very different problems. First, an ML task: he pointed his agent at the repo and asked it to build a version for his qmd query-expansion model. After 8 hours and 37 experiments he woke up to a 19% higher score on a 0.8B model, beating his previous 1.6B model. “I’m not a ML researcher of course,” he added.

Then he tried something more interesting. Using Shopify’s general-purpose port (below), he ran it on the Liquid templating engine, the rendering system behind Shopify storefronts: 53% faster combined parse and render time, 61% fewer object allocations. This was not ML research. It was performance engineering on production code, and Lutke flagged the catch himself: “This is probably somewhat overfit, but there are absolutely amazing ideas in this.”

SkyPilot: Scaling to 16 GPUs for $300

The SkyPilot team parallelized AutoResearch across 13 H100s and 3 H200s on two Kubernetes clusters, with Claude Code as the agent. In about 8 hours they submitted ~910 experiments (~700 with valid results), roughly 9x the throughput of a single-GPU sequential run.

The total cost was under $300: about $9 in Claude API calls and about $260 in GPU compute. val_bpb improved from 1.003 to 0.974, a 2.87% gain, and returns flattened to almost nothing after ~700 experiments.

One detail stood out. Nobody told the agent the GPUs differed in speed, yet it developed a two-tier strategy: screen ideas on the H100s, then confirm the top two or three candidates on the faster H200s.

What Happened to AutoResearch After March 2026

The repo: frozen on purpose

The karpathy/autoresearch repo is public, MIT-licensed, and effectively finished. The last commit landed on March 26, 2026; since then it has kept collecting stars (about 98,000 by late October) and forks (about 13,600), with nearly 200 open issues and pull requests. Karpathy has said he does not want to bloat it with CPU or Apple Silicon support, so the README points to notable forks instead: macOS, MLX, Windows RTX and AMD. If you want a running list of everything built on the idea since, awesome-autoresearch (about 2,500 stars) is the most complete index.

Karpathy’s follow-up: from one PhD student to a research community

The day after launch, Karpathy wrote that the next step “has to be asynchronously massively collaborative for agents (think: SETI@home style). The goal is not to emulate a single PhD student, it’s to emulate a research community of them.” GitHub, he argued, is “almost but not really suited for this” because it assumes one master branch.

His prototype was AgentHub: one Go binary, a SQLite database, a bare git repo and a message board, with “no main branch, no PRs, no merges,” just “a sprawling DAG of commits going in every direction.” The README called it “Work in progress. Just a sketch.” The original karpathy/agenthub repo now returns a 404; the code survives in community copies such as ottogin/agenthub. Karpathy has not published a successor repo since.

On the No Priors podcast on March 20 he laid out the limits plainly (“if you can’t evaluate it, then you can’t auto research it”) and the missing piece for a swarm: “a system where an untrusted pool of workers can collaborate with a trusted pool of workers that do the verification.” The community version that got closest is autoresearch-at-home, which lets agents on different machines claim experiments, publish every result with its full train.py, and pull a shared global best.

Shopify’s pi-autoresearch: the general-purpose version

The most widely used descendant is not an ML tool. pi-autoresearch, built by Shopify engineer David Cortés and Tobi Lutke, packages the keep/discard loop as an extension for the pi coding agent and works on any target you can measure: test speed, bundle size, build times, Lighthouse scores, or LLM training. It has about 8,200 stars. Shopify’s April write-up says it improved 40+ metrics internally, including a build that got 65% faster after the agent noticed a TypeScript transform processing all 580 component files when only 105 needed it, unit tests running 300 times faster, and React components mounting 20% faster.

Bigger swarms on the same benchmark

In September, Rekursiv published “(auto)²-research,” in which role-based teams of agents share a graph of hypotheses and failed experiments. Over 6,164 experiments in about three days they report 0.887791 mean val_bpb across 10 seeds, measured on a B200 rather than an H100, so it does not compare directly with the numbers above. The authors also note that humans set priorities and that several gains were never isolated.

Research papers that now use AutoResearch as the baseline

The clearest sign of influence is academic. Within seven months, “autoresearch” became a generic name for the propose, run, keep-or-revert loop, and Karpathy’s repo became the standard baseline:

PaperWho, whenWhat it changesResult
AutoKernelRightNow AI, March 2026Same loop, applied to Triton and CUDA kernels5.29x faster RMSNorm than PyTorch eager on H100
GEARMay 2026Population-based genetic search instead of one incumbentBest variant beat the baseline’s 100-experiment result (0.98232 bpb) by experiment 40
RehearseTencent, July 2026Ranks proposals before spending a training run, plus focused memory10.7% vs 7.1% mean improvement on nanochat over 100 thirty-minute experiments, 5 seeds
An AI Scientist that Doesn’t DriftCaltech, July 2026Immutable experiment cards, a “taste” oracle, parallel research streamsBuilt because their autoresearch loop “drifted toward refinements of the current best configuration”
Fork-and-FlushMicrosoft Research, October 2026Forks the agent into fresh-context copies, continues from the best66% relative gain over a single run at equal compute, across 13 long-horizon tasks

The Criticism, Seven Months In

It is a greedy hill-climber. Someone opened issue #22, “Low creativity”, on launch day. The papers above make the case in detail: GEAR calls the loop “effectively single-incumbent hill climbing” that discards partial ideas and settles into one local optimum, and Microsoft Research found that independent runs of the same agent on the same task “plateau at substantially different scores,” stuck in what they call idea basins. The ratchet never takes a step back to set up a bigger gain later.

The guardrails are prose. Issue #599, opened in May and still open, shows four ways an agent could lower the printed val_bpb without a better model: skip the optimizer while still burning the time budget, wrap or replace the evaluate_bpb call (the ban on touching it lives only in program.md), shrink the eval set, or print a number nothing binds to the actual weights. The proposed fix is a 30-line log hashing code, data and final model state next to each result. None of this means Karpathy’s runs were gamed: he re-tested his round-1 changes on bigger models, and the leaderboard times come from full 8xH100 training runs. It means an unattended loop can only be trusted as far as its harness is enforced.

Small gains are noisy and hardware-bound. Each keep or discard decision rests on one five-minute run. Some kept changes in the overnight log Karpathy’s agent posted improved val_bpb by less than 0.001. SkyPilot found that rankings flipped between GPU types (FINAL_LR_FRAC=0.03 beat 0.05 on H100 and lost on H200). Lutke called his Liquid results “probably somewhat overfit.” The teams that report the most convincing numbers now run multiple seeds, as Rehearse and Rekursiv do.

“Isn’t this just AutoML?” Critics quoted by Fortune said Karpathy had rediscovered part of AutoML, which big labs have used for years. His reply: neural architecture search “is such a weak version of this,” and “It’s not even close.” Classic NAS and AutoML search a predefined space (a range of learning rates, three activation functions, four layer counts) with random or evolutionary methods. AutoResearch searches over code: the agent that noticed QKNorm lacked a scaler multiplier was reading the script and spotting an oversight, not sampling a grid. The fair version of the critique is narrower. The keep-or-revert loop is old, the proposal step is new, and the proposals are only as good as the model writing them.

It depends on the agent harness. Karpathy filed issue #57 himself: Codex “ignores instruction to never stop,” while Claude kept going. The loop is only as good as the agent’s willingness to run unattended for hours.

You are told to switch off the safety rails. The README says to start your agent “and disable all permissions.” An agent with unrestricted shell access running overnight belongs in a container or VM, not on your workstation.

The Karpathy Loop as a Design Pattern

Analyst Janakiram MSV coined the term “The Karpathy Loop” for the pattern that makes AutoResearch work. It has three requirements:

  1. One modifiable file that the agent controls completely
  2. One objectively testable metric that determines success or failure
  3. A fixed time budget per experiment to prevent runaway costs

Any problem that can be structured this way can be autoresearched. The metric does not have to be validation loss. It could be page load time, memory usage, rendering speed, test runtime or build time, which is exactly where Shopify ended up.

Karpathy made this explicit in his results thread: “any metric you care about that is reasonably efficient to evaluate (or that has more efficient proxy metrics such as training a smaller network) can be autoresearched by an agent swarm.” He also predicted that “All LLM frontier labs will do this. It’s the final boss battle.”

Related: AI Agent Benchmarks Explained: What SWE-bench, WebArena, and AgentBench Actually Measure

What Automated Research Means for Scientific Publishing

AutoResearch’s output is a git branch and a results.tsv. Every claim is a commit you can re-run, and every failure is logged next to the wins. That is its quiet strength, and it is worth remembering as the ecosystem moves from optimizing code to writing it up. Karpathy imagined agents contributing “a little ‘paper’ of findings back” to a shared pool, and pipelines like AutoResearchClaw already turn a single idea into a full conference-style paper.

Publishing is already absorbing machine-shaped prose: a study that scored the full text of about 12,750 arXiv papers from 2021 to 2026 with an academic-writing AI detector found that over 30% of recent papers read as machine-written, with a clear upward trend since late 2022 and computer science highest at about 65%, against a flag rate of about 0.4% on pre-ChatGPT papers that serves as the detector’s false-positive rate. (Disclosure: unslop is run by the author of Paperclipped.) The study is careful about what that number means. “Reads as AI-written” is not the same as “AI did the research”: a flag cannot tell a fully generated paper from one an LLM merely edited, and the author warns that a single score should never be used to accuse anyone. The detector also misses roughly a fifth of AI-written papers and may do worse on notation-heavy fields like mathematics, so the true share is probably higher and the per-field baselines are noisy.

Put those two trends together and the conclusion is uncomfortable for anyone who reviews papers: prose is becoming the cheapest part of research, and the least informative. What separates a real result from a plausible one is the artifact underneath, meaning the code, the seeds and the log of what failed. The best follow-ups to AutoResearch are moving in that direction, from Caltech’s immutable experiment cards (“so a falsified hypothesis cannot be retconned”) to the integrity log proposed in issue #599.

Related: AutoResearchClaw: The 23-Stage AI Pipeline That Writes Conference Papers from a Single Idea

Running AutoResearch Yourself

You need a single NVIDIA GPU (the repo is tested on H100), Python 3.10+, and the uv package manager. On Macs, Windows RTX cards or AMD GPUs, start from one of the notable forks listed above instead.

git clone https://github.com/karpathy/autoresearch
cd autoresearch
uv sync
uv run prepare.py   # one-time: download data, train the tokenizer (~2 min)
uv run train.py     # one manual 5-minute run to confirm the setup works

Then start Claude Code, Codex or another coding agent in the repo and use the README’s kickoff prompt: “Hi have a look at program.md and let’s kick off a new experiment! let’s do the setup first.” The agent creates a branch like autoresearch/oct23, records a baseline, and starts looping.

On smaller GPUs, the README suggests a lower-entropy dataset such as TinyStories, a smaller vocab_size, a shorter MAX_SEQ_LEN, fewer EVAL_TOKENS, DEPTH down from 8 to around 4, and a WINDOW_PATTERN of just “L”.

What a night looks like. Karpathy’s agent posted a summary of one overnight run as Discussion #43: about 10.5 hours on an H100, 126 experiments, 23 kept, 102 discarded, 1 crash, with val_bpb going from 0.9979 to 0.9697. The single biggest win was halving the batch size, which buys more optimizer steps inside the five-minute window. The biggest failure, tying the input and output embeddings, made the model 2.24 bits per byte worse. Expect most experiments to be discarded. That is the loop working, not failing.

Which agent. In March, Claude models were the reliable choice: Latent Space relayed a report of Claude Opus 4.6 running 12+ hours and 118 experiments without breaking the loop, while GPT-5.4 at its highest reasoning setting failed to follow the “LOOP FOREVER” instruction. Whatever you use, the test is simple: if it stops after three experiments to ask whether it should continue, it is the wrong tool for this.

Cost and safety. GPU time is the real bill. SkyPilot’s 910 experiments cost about $9 in Claude API calls and about $260 in compute. Run the agent in a sandbox, since the README tells you to give it every permission. And if your metric is not validation loss, skip the fork and use pi-autoresearch, which is built for arbitrary benchmarks.

Related: AI Agent Sandboxing: MicroVMs, gVisor, and WASM for Safe Code Execution

Frequently Asked Questions

What is Karpathy’s AutoResearch?

AutoResearch is an open-source repo by Andrej Karpathy that lets an AI coding agent run ML research experiments on its own on a single GPU. The agent edits a 630-line training script, trains for exactly five minutes, and keeps or reverts each change based on validation bits per byte (val_bpb). Released in March 2026, it had about 98,000 GitHub stars by October 2026.

Is AutoResearch still maintained?

The repo is public and MIT-licensed, but it has had no commits since March 26, 2026. Karpathy treats it as a minimal seed. Development happens in forks for macOS, MLX, Windows and AMD, in Shopify’s general-purpose pi-autoresearch, and in research projects such as GEAR, Rehearse and Fork-and-Flush that use it as a baseline. His multi-agent follow-up, AgentHub, was a sketch and is no longer public.

What results has AutoResearch produced?

Karpathy’s first run, about 700 experiments over two days, found roughly 20 stackable improvements that cut nanochat’s time to GPT-2 from 2.02 to 1.80 hours. A second round brought it to 1.65 hours. Tobi Lutke got a 19% better score from a 0.8B model in 37 experiments, and Shopify’s port made Liquid parse and render 53% faster. SkyPilot ran about 910 experiments on 16 GPUs in 8 hours for under $300.

What hardware do you need to run AutoResearch?

The original repo needs a single NVIDIA GPU (tested on H100), Python 3.10+ and the uv package manager. Community forks listed in the README support macOS, Apple MLX, Windows RTX cards and AMD GPUs. On smaller machines, the README recommends the TinyStories dataset, a smaller vocabulary, shorter sequences and a lower model depth.

What are the main criticisms of AutoResearch?

The loop is greedy hill climbing, so runs get stuck in local optima that papers like GEAR and Fork-and-Flush try to escape. The rules against touching the evaluation are instructions rather than enforcement, so an agent could game val_bpb. Single five-minute runs make small gains noisy and hardware-specific. Critics also call it a rediscovery of AutoML, though it searches over code rather than a fixed parameter grid.