About the benchmark

Harvey LAB turns legal assignments into file-based tasks. Agents read matter documents and instructions, then produce contracts, memoranda or due-diligence analyses that judge models assess against individual criteria.

Released
Created by
Harvey
Task set
2,010 public tasks across 27 practice areas

Core capabilitiesLegal analysis, cross-document retrieval, planning and document delivery

More information

Domains and task mix

Assignments cover practice areas including corporate M&A, intellectual property, private equity and venture capital, governance, litigation and real estate. They include drafting, review, diligence, retrieval and analysis, with unequal task counts across areas.

Task sources and creation

Task design starts from matters and assignments encountered by practicing lawyers. Document and scenario-generation workflows create simulated materials, paired with expert-written criteria. The repository uses synthetic matter files for evaluation.

Inputs and deliverables

Inputs include short instructions and a directory of contracts, emails and other matter materials. Agents use file and terminal tools, then write the required deliverables to an output directory for grading.

A representative task

An M&A example asks for a review of change-of-control clauses, contractual provisions triggered when ownership changes. The agent identifies relevant contracts, assesses transaction risks, consolidates consent or waiver requirements and delivers a memorandum for the deal team and board.

Test conditions

Each task runs in a separate Podman container, an isolated workspace. This version disables task-environment networking, mounts source documents read-only and permits writes to the output directory. Run settings specify the model and response-turn limit.

Scoring

Judge models mark each criterion pass or fail. For one judge, every criterion must pass for the task to pass. This version also supports two independent judges: its dual-judge rate averages their outcomes, while a separate field records agreement on passing every criterion. Criterion-level pass rates are retained.

Abstract

The stratified sample covers 100 tasks from the 2,010-task public release, with 20 Accepted, 9 Doubtful and 71 Refused.

Confirmed defects mainly involve erroneous grading references and requirements not disclosed to the solver. We examine whether scoring accurately assesses the intended task.

Review outcomes
OutcomeTasksMeaning
Accepted20Within the scope of this review, we found no issue affecting evaluation validity and no specific concern awaiting verification.
Doubtful9We found a specific concern whose trigger, correct interpretation or effect on scoring still needs verification.
Refused71We confirmed at least one issue affecting evaluation validity; its actual effect depends on the relevant trigger conditions.
Total100

Scope and version

We reviewed the Harvey LAB public release, 4743403f52c1, sampled by practice area and work type. Findings are current to November 6, 2026.

Issue distribution

Distribution of findings
Issue typeRefusedDoubtfulTotal
C1Incorrect answers or scoring logic36339
C2Scoring requirements not stated in the task30535
C3Missed critical errors415
C5Other issues101
Total71980

Each task is counted under one primary cause. Each of the 71 Refused tasks contains at least one material defect, whose impact depends on the affected scoring criteria.

Representative cases

Annual compliance costs

C1Incorrect answers or scoring logic

Task: corporate-governance/summarize-state-ai-governance-bill-for-product-and-engineering-teams Refused

The task asks the model to summarize an AI-governance bill and calculate a company’s annual compliance costs. The materials list two expenses: a $60,000 assessment update every year and a $120,000 audit every two years.

The annual average is therefore $60,000 + $120,000 / 2 = $120,000. The rubric ignores the two-year audit cycle and instead expects $60,000 + $120,000 = $180,000.

The actual response gives the correct $120,000 average and explains that costs are $180,000 in audit years and $60,000 in the other years. This criterion still fails it. The grading reference penalizes the correct annual calculation.

Evidence

An emoji absent from the source

C2Scoring requirements not stated in the task

Task: employment-labor/extract-labor-employment Refused

The task asks the model to extract facts from employment-dispute materials, including a Slack conversation. Its job is to report what the supplied records contain.

The rubric requires mentioning a wink emoji in that conversation, but the emoji is absent from the original attachments supplied to the model. The response faithfully extracts the event and still fails this criterion for omitting the emoji.

Evidence

An ungraded drafting memo

C3Missed critical errors

Task: contracts/banking/account-control-agreement-first-draft/scenario-02 Refused

The task requires two files: a bank account-control agreement and a separate drafting memo. The memo must identify departures from the client playbook’s preferred terms, while the work must flag terms at minimum acceptable risk thresholds.

All 51 criteria check agreement terms and notice exhibits; none checks the drafting memo. An agreement that meets those criteria can satisfy every listed condition for full credit even if the memo is never submitted.

The client explicitly asks to know which terms need attention, yet the scoring conditions allow that entire risk explanation to be missing. They do not distinguish this incomplete delivery from one containing both required files.

Evidence

Conflicting price constraints

C5Other issues

Task: contracts/corporate-ma/joint-venture-agreement-term-negotiation/scenario-01 Refused

The task asks the model to draft drilling-overhead terms acceptable to both companies without exceeding either party’s negotiation authority.

One party caps the base rate at $70,000 per well per month and requires at least a 15% discount for the second and subsequent wells drilled simultaneously on the same pad. The other requires the discounted rate to remain at or above $60,000 per well per month in all circumstances.

Even the maximum $70,000 rate becomes just $59,500 after a 15% discount—below the other party’s $60,000 floor. A lower base rate only widens the gap.

No price satisfies both mandatory conditions. Yet the task still requires contract-ready compromise clauses within both parties’ limits, leaving no passing route for correctly identifying the conflict and requesting revised authority.

Evidence

Per-task results

Refused: 71 tasks

Doubtful: 9 tasks

Evaluation impact and updates

These issues can deny models credit for accurate reading and reasonable drafting choices. Correcting a single erroneous criterion may not change an overall score; each issue’s effect on actual scoring depends on the relevant trigger conditions.

Author clarifications, counterevidence and fixes will be assessed and reflected in task-level updates.