Open Athena: posttraining has no Common Crawl, and SWE-bench shows why

Pretraining had a commons. Posttraining does not. That is the argument David Hall makes in a new Open Athena explainer published October 15, and the part worth sitting with is not the thesis. It is the arithmetic. The same essay that frames the problem elegantly also documents, in passing, how brutal the conversion from human work to reinforcement-learning task actually is, and the numbers it cites are the strongest case against the industry’s current approach that anyone has assembled in one place.

The distinction Hall draws is clean: “Pretraining consumes documents, while posttraining consumes tasks.” Documents were lying around. Common Crawl has archived roughly 10 petabytes since 2007. GPT-3 was trained on about 60% Common Crawl-derived data, and the original Llama mix was almost entirely extracted from the internet commons. Nobody had to pay Wikipedia editors to write an encyclopedia so that a model could read it. They wrote it for their own reasons.

Tasks are different. A human task is situated: someone wants something, acts, the world responds, the goal shifts, and then the moment is gone. Hall compares this to oral culture, where a story is an event as much as an object. Writing it down freezes one telling. Encoding it as an RL task does the same thing, and the freezing is the expensive part.

The SWE-bench attrition

Software is the easy case, Hall says, because software engineering is already the practice of writing tasks down. Version control, continuous integration, unit tests, containers: decades of infrastructure for preserving a world, an intent, and a record of actions. If any domain should convert cleanly into RL tasks, it is this one.

Then he walks through what actually happened. SWE-bench’s builders crawled GitHub issues and pull requests from 12 Python repositories, yielding a bit over 90,000 PRs. Only about one in eight mapped readily to the frame. The rest were unmerged, unlinked to an issue, or did not change a test. Of the remaining 11,407, about 80% could not be turned into executable tasks, because the package could not be reconstructed at that state or the runtime misbehaved. That left 2,294 tasks.

A year later OpenAI hired 93 experienced Python developers to review 1,699 randomly selected SWE-bench tasks for SWE-bench Verified. More than two thirds were flagged for removal: underspecified prompts, unfair tests, other major issues. The filter produced 500 tasks. A later audit found another sixth or so of those 500 were broken anyway.

Hall is careful not to dunk on SWE-bench. He calls it one of the better constructed public coding datasets available. That is the point. The best-case artifact in the best-case domain, built by motivated people, lost more than 97% of its raw material on the way to a usable benchmark.

It gets worse outside the easy case

The essay piles on. In TaskTrove, a subset of 5,000 converted Java tasks saw more than 90% discarded during repair: tests that did not parse, checked nothing, or leaned on missing project context and external services. In a Python subset, almost 98% of inspected verifier bundles could not even start their tests because dependencies were missing. An audit by Vals found two thirds of the 2,698 coding environments in Xiaomi’s MiMo v2.6 public release still contained reference fixes recoverable from Git. In one demonstrated case, the agent copied the fix and passed all five hidden tests.

That last one matters more than the others. A broken task is waste. A leaky task is worse than waste, because it trains an agent to find the answer key rather than do the work, and the benchmark reports success while the capability stays flat. Every one of those 2,698 environments was presumably shipped with some confidence.

If the best-case artifact in the best-case domain loses more than 97% of its raw material, the rest of the economy has no chance of converting cleanly.

What the frontier labs are actually buying

Hall’s answer to “what’s a frontier lab to do” is blunt: pay the lawyers, scientists, salespeople, and doctors to fit their work into the RL task frame. Build the tasks, environments, rubrics, verifiers, simulators. Coordinate and verify all of it. That is the raison d’être of the posttraining data industry, and it explains a lot of the hiring and vendor spend that looks strange from the outside. The labs are not buying data. They are buying inscription labor.

For the open models community, the implication is harsher. There is no Common Crawl for posttraining because there is nothing to crawl. Hall notes that modern open LLMs like Nemotron still lean on the web commons for the vast majority of pretraining, and Cloudflare reports that over half of crawler traffic is now for AI. But the pretraining commons is a legacy asset. The posttraining commons has to be built, and the people building it are being paid by whoever can afford the invoice.

Hall points at GDPval as one model for doing this systematically, and floats the possibility of a better paradigm entirely. He is honest that the alternative is grim: a future where the inscriptions of human work become private assets held by a few companies.

The take

Tessera’s read is that the SWE-bench numbers should reframe how the industry talks about posttraining data quality. The current conversation treats task construction as a curation problem, something a good pipeline and a careful filter can solve. Hall’s evidence says it is a manufacturing problem, with a yield rate, and the yield rate on the easiest possible input is somewhere near 3%. Anyone quoting a task count without a discard rate is quoting a number that means less than it appears to.

The second implication is about benchmark trust. If SWE-bench Verified, assembled by 93 paid reviewers, still had roughly a sixth of its tasks broken after the fact, then the smaller, cheaper, faster benchmarks that labs cite in model cards deserve proportionally more skepticism, not less. The Vals finding on MiMo v2.6 is the template: reference fixes sitting in the Git history, agents copying them, hidden tests passing. That failure mode is cheap to introduce and expensive to detect.

For builders, the practical question is where the open ecosystem gets its task supply. Hall’s suggestion that agents can help reconstruct tasks from outputs and traces is the most actionable line in the piece, and also the least proven. An agent that inscribes tasks is doing the same work that just failed at 97% when humans did it by hand. Whether the automation improves the yield or just accelerates the discard is an open question, and it is the one that decides whether an open posttraining commons exists at all.

The posttraining data industry, meanwhile, is already pricing that question in. The labs paying doctors and lawyers to formalize their work are not waiting for an answer.