Harvey's open-source LAB benchmark puts legal agents on the stand

Harvey, the legal AI company valued at over $3 billion in its 2025 Series E, has open-sourced a benchmark for legal agents called LAB, the Legal Agent Benchmark. The project, released under the harveyai/harvey-labs repository, pairs a dataset of legal tasks with an execution harness that runs agents against those tasks and scores them. It is an unusual move for a company whose core product is proprietary, and it signals something specific: the bottleneck in legal AI is no longer model capability, it is evaluation.

The benchmark’s structure is straightforward. LAB consists of two parts. First, a dataset of tasks containing agent instructions, documents, and rubrics. Second, an execution harness for running and evaluating agents against those tasks. The documentation describes a full walkthrough in docs/tutorial.md that takes one realistic M&A data-room assignment end to end: setup, task inspection, agent run, scoring, report review, and comparison dashboards. The architecture page covers the task model, harness, tools, adapters, reports, and sweeps. The evaluation methodology section describes “all-pass rubric scoring and LLM judge behavior.”

That last detail is the part worth reading twice. All-pass rubric scoring means an agent does not get partial credit. Every criterion in the rubric must be satisfied or the task is scored as failed. This is a deliberate departure from the F1-score and pass@k conventions that dominate general-purpose agent benchmarks. In legal work, partial credit is malpractice. A contract analysis that misses one material term is not 90% correct; it is a liability.

The choice of an LLM judge is also notable. LAB uses LLM judges to evaluate agent outputs against rubrics, which raises an obvious question: if the judge is an LLM, and the agent is an LLM, what exactly is being measured? The answer, per the documentation, is that the rubrics are designed to be objective and checkable. The judge is not grading quality on a vibe scale. It is checking whether specific, enumerated conditions hold. That design choice matters because it makes the benchmark more reproducible and less susceptible to the judge-model collapse problem that has plagued other LLM-as-judge evaluations.

Harvey’s motivation for releasing this is not pure altruism. The company has a commercial interest in a standardized evaluation layer for legal agents. Every law firm evaluating Harvey’s product, or a competitor’s, currently runs its own ad hoc testing. A shared benchmark creates a common vocabulary for procurement, and Harvey gets to define that vocabulary. The citation format in the repository, with the announcement post linked from harvey.ai, makes the marketing intent explicit. This is a benchmark, but it is also a moat.

The timing matters. Legal AI is the most commercially active vertical for agentic systems. Harvey’s competitors include CoCounsel from Thomson Reuters, Lexis+ AI from LexisNexis, and a wave of startups like Robin AI and Spellbook. All of them claim their agents can handle due diligence, contract review, and litigation support. None of them, until now, had a public, reproducible way to test those claims. LAB is the first credible attempt at a standard, and Harvey gets to set the terms.

What LAB does not do is measure the things that make legal work hard in practice. The benchmark tasks are realistic, but they are static. They do not capture the adversarial nature of legal practice, where opposing counsel is actively trying to hide or obscure relevant information. They do not capture the client relationship, where the deliverable is shaped by a partner’s preferences. And they do not capture the ethical dimension, where an agent must know when to stop and ask a human. Those are the hard parts of legal work, and no benchmark can fully simulate them.

The all-pass rubric design partially addresses this. By requiring every criterion to be met, LAB forces agents to be conservative. An agent that hallucinates a case citation fails even if it correctly identified the other forty issues. That is the right incentive for legal work. The risk is that benchmark overfitting pushes agents toward rubric-gaming, where the model learns to satisfy the checklist without doing the underlying analysis. Harvey’s documentation acknowledges this by describing LAB as “an ongoing project” with plans to “consistently add to and refine the task set and execution harness.”

There is a deeper implication for the broader agent ecosystem. Harvey LAB is one of the first vertical-specific agent benchmarks from a company that is itself a benchmark subject. The pattern is familiar from other domains. OpenAI released SWE-bench for coding agents, and now every coding agent claims a SWE-bench score. Anthropic released the OSWorld benchmark for computer-use agents. The vertical benchmark is becoming a competitive weapon, and Harvey is deploying it in legal.

The open-source choice is the shrewd part. Harvey could have kept its evaluation stack internal, using it only to tune its own models. By releasing it, Harvey gains three things: external validation of its approach, a recruiting signal for AI researchers who want to work on legal evaluation, and a gravitational pull on the entire legal AI ecosystem toward its definition of quality. The cost is that competitors can use LAB to improve their own agents, but that cost is acceptable because the benchmark favors the team that builds the best agents, and Harvey believes that is itself.

The LLM judge component deserves scrutiny. The documentation describes the judge behavior in detail, but the specifics of which models are used as judges, and how judge agreement is measured, are not fully specified in the repository readme. A benchmark whose judge is a black box is a weaker benchmark. Harvey should publish judge agreement statistics and error analysis. Without that, a skeptic cannot distinguish between an agent that solved the task and a judge that was fooled.

What LAB gets right is the unit of analysis. It evaluates agents, not models. The harness runs the full agent loop: reading instructions, pulling documents from the data room, calling tools, drafting the report, and self-checking. This is the correct abstraction for legal work, where the value is in the end-to-end execution, not in the raw reasoning of a single model call. The M&A data-room walkthrough in the tutorial is a good example because it requires multi-step tool use and document synthesis, not just text generation.

For AI builders outside legal, the lesson is transferable. The all-pass rubric design is applicable to any domain where errors are asymmetric, where a single mistake is disproportionately costly. Finance, healthcare, and compliance all have this property. The pattern of a vertical company releasing a benchmark to define its market is also transferable. Expect to see similar releases from companies in those sectors within the next year.

The open question is whether LAB becomes the standard or just another benchmark in a crowded field. The legal AI market has room for one evaluation standard, and Harvey has the brand, the customer base, and now the open-source artifact to claim it. The repository is fresh, versioned at v1.0, and structured for contribution, with a documented process for adding tasks and model adapters. That is the right foundation.

Harvey’s LAB is a benchmark, but it is also a statement of intent. The company is saying that legal AI has moved past the demo phase and into the measurement phase. The firms that buy legal AI will now have a common yardstick, and the agents that score well on all-pass rubrics will be the ones that win the procurement cycles. The rest of the industry will be measured against a standard that Harvey defined. That is the real news here, and it is worth more than the benchmark itself.