Nearly half of 60 AI benchmarks are saturated, and expert curation is the fix

A 37-author study accepted at ICML 2026 has quantified what every AI researcher has suspected for years: benchmarks die. The paper, “When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation”, analyzes 60 language model benchmarks across 14 saturation-related properties and finds that nearly half exhibit saturation, with rates increasing as benchmarks age. The surprising part is what does not save them. Public test data does not protect a benchmark from saturation. Expert curation does.

The finding cuts against a decade of evaluation orthodoxy. Since the ImageNet era, the field has treated test-set leakage and public exposure as the primary threat to benchmark validity. This study, led by Mubashara Akhtar and including Stella Biderman, Sanmi Koyejo, Irene Solaiman, and Zeerak Talat, suggests the mechanism of death is more mundane: models simply get good enough that the benchmark stops discriminating between them. Saturation is not a security failure. It is a measurement failure.

The paper defines saturation formally and applies that definition across benchmarks of varying age and provenance. The authors track 14 properties, from test-set publicity to expert curation, and test which ones correlate with resilience. The headline result is stark. Nearly half of the 60 benchmarks studied are saturated, and the probability of saturation climbs with benchmark age. Old benchmarks are not just less useful. They are actively misleading, because a model that scores 98% on a saturated benchmark looks indistinguishable from a model that has genuinely mastered the underlying capability.

What distinguishes the survivors is instructive. Expert curation, meaning benchmarks assembled and maintained by domain specialists rather than scraped or crowd-sourced, correlates strongly with longevity. Public test data does not. A benchmark with a public test set is no more resilient than one with a private set. The field has spent enormous effort building private test sets, gated leaderboards, and hidden evaluation suites, on the theory that secrecy preserves usefulness. This study says that effort is misdirected. The design of the benchmark matters more than its secrecy.

This is a genuinely useful result, and it arrives at a moment when the evaluation ecosystem is visibly straining. The past 18 months have produced a parade of leaderboards that collapse into noise within weeks of release. The MMLU successor race, the reasoning-benchmark arms race, the agentic-evaluation scramble, all follow the same arc: release, saturate, retire. The paper gives the field a vocabulary for that arc and, more importantly, a design principle for escaping it.

The expert-curation result deserves scrutiny before the field adopts it as gospel. Correlation is not causation, and the paper’s 60-benchmark sample, while substantial, skews toward well-known English-language benchmarks. Expert curation may correlate with other longevity factors, such as a benchmark’s ability to evolve through new task versions or its grounding in a stable domain like mathematics or code. A benchmark curated by experts may simply be better designed in ways the 14 properties do not capture. The authors acknowledge this in their framing, treating the result as a design signal rather than a settled law.

Still, the implication for benchmark builders is concrete. If expert curation is the resilience factor, then the field should invest in benchmarks that are living artifacts, maintained by people who understand the domain, rather than static datasets released once and abandoned. This aligns with a broader shift already underway in the industry. Anthropic, OpenAI, and Google all maintain internal evaluation teams that treat benchmarks as ongoing programs rather than one-time releases. The paper’s contribution is to show that this practice is not just corporate hygiene. It is the difference between a benchmark that lasts and one that dies.

For AI builders, the practical takeaway is uncomfortable. Leaderboard scores are decaying assets. A model’s MMLU or HumanEval number, once the coin of the realm for model comparison, now tells you less about the model than about the benchmark’s age. Purchasing decisions, open-source model selection, and research prioritization based on saturated benchmark deltas are decisions made on noise. The paper suggests replacing single-number comparisons with evaluation suites that are continuously refreshed and domain-expert-maintained.

The policy angle is quieter but real. Regulators and standards bodies, from the NIST AI Safety Institute to the EU AI Office, rely on benchmarks as objective evidence of model capability and risk. If roughly half of the benchmarks in a systematic sample are saturated, then any regulatory regime built on static benchmark thresholds is building on sand. The paper does not address regulation directly, but its findings undermine the premise that a benchmark score is a stable, comparable measurement across time. A model that scores 95% on a saturated safety benchmark may be no safer than one scoring 80%.

The field has been here before. ImageNet saturated, and the community responded with harder benchmarks, then with robustness suites, then with a recognition that no single benchmark captures generalization. The current study suggests the response cycle itself needs updating. Instead of waiting for saturation and then building the next benchmark, the field should build benchmarks that are designed to resist saturation from the start, through expert involvement and continuous maintenance.

The paper’s most provocative line is buried in its abstract: resilience to saturation is impacted by expert-curation, not by public test data. That sentence, if it holds up to replication, inverts a decade of evaluation security thinking. The private-test-set industry, the gated leaderboards, the API-only evaluation services, all built on the premise that hiding the test data preserves its value, may have been solving the wrong problem. The problem was never exposure. It was design.

What to watch now is whether the ICML acceptance translates into practice. Benchmark builders rarely redesign their benchmarks in response to a single study, and the incentives to release new benchmarks, with their attendant publicity, are stronger than the incentives to maintain old ones. The paper offers a path, but the field must choose to take it. The alternative is a future where every benchmark is a tombstone, marking where model capability used to be, and leaderboards are museums of obsolete measurements. The study at least tells us which tombstones last longer.