ContractScrub: A benchmark for final review of legal contracts

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Yejin Bang, Kirsty Fielding, Brandan Oliver, Brian Birke, Nabeel Seedat, Andrew M. Bean
• Venue: arXiv: cs.AI
• Published: 2026-08-20
• Collected: 2026-08-21
• URL: http://arxiv.org/abs/2608.20204v1
• PDF: https://arxiv.org/pdf/2608.20204v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

Researchers introduce ContractScrub, a new benchmark to evaluate the performance of language models in contract scrubbing, which is the final review of legal contracts for errors and inconsistencies. The study finds that current language models perform poorly in this task, highlighting the need for domain-specific benchmarks.

What the Document Says

• The text discusses the use of language models (LLMs) in legal work, particularly contract scrubbing.
• Contract scrubbing is described as a routine task requiring detailed attention to long documents and involving long-context reasoning, consistency checking, and named entity recognition.
• Despite the potential for automation and economic value, no formal evaluations of LLMs performing contract scrubbing have been conducted.
• The researchers introduce ContractScrub, a benchmark designed for evaluating contract scrubbing capabilities.
• ContractScrub comprises contracts hand-crafted by experienced lawyers over diverse error categories such as misuse of defined terms, incorrect references, and inconsistent language.
• Only one model reached a macro average recall of 0.75 in the ContractScrub benchmark despite strong performance on seemingly related general benchmarks.
• This finding suggests practical limits of current models and the importance of domain-specific benchmarks for measuring real-world impact.

Key Concepts Mentioned

• language model
• contract scrubbing
• benchmark
• long-context reasoning
• consistency checking
• named entity recognition
• macro average recall

Claims Made by the Document

• Contract scrubbing is a suitable task for automation.
• Frontier models perform poorly in contract scrubbing.
• Current models have practical limits.

Evidence Used

• heavy reliance on processing large amounts of text
• routine and painstaking work requiring detailed attention to long documents
• only one model reaching 0.75 macro average recall

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Legal work, with its heavy reliance on processing large amounts of text, is often considered one of the domains most exposed to the use of LLMs. Contract ``scrubbing,'' the final review of transactional agreements for errors and inconsistencies, is a particularly suitable task for automation, because it is routine, painstaking work requiring detailed attention to long documents. Scrubbing also seems to align naturally with the general capabilities expected of frontier LLMs around long-context reasoning, consistency checking, and named entity recognition (NER). Despite the economic value and potential for automation, no formal evaluations of LLMs performing contract scrubbing have been conducted. We introduce ContractScrub, the first benchmark designed to evaluate contract scrubbing capabilities, comprising contracts hand-crafted by experienced lawyers over diverse error categories such as misuse of defined terms, incorrect references, and inconsistent language. Frontier models perform surprisingly poorly with only one model reaching 0.75 macro average recall despite strong performance on seemingly related general benchmarks, demonstrating the practical limits of current models and the importance of narrowly targeted, domain-specific benchmarks for measuring real-world impact.