InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Samuel J. Vincent, Daniel Calloway, Fangyi Yu, Andrew M. Bean, Nabeel Seedat
• Venue: arXiv: cs.AI
• Published: 2026-08-20
• Collected: 2026-08-21
• URL: http://arxiv.org/abs/2608.20220v1
• PDF: https://arxiv.org/pdf/2608.20220v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

Researchers introduce InsufficiencyBench, a new legal benchmark to evaluate how well AI models can identify when user queries lack important information needed for accurate legal advice. The study finds that current AI models struggle with this task, often either making indiscriminate hedges or giving silent answers based on false assumptions.

What the Document Says

• The text discusses a study about the evaluation of legal AI systems.
• The study focuses on legal AI systems answering legal questions where queries arrive without fully specified information.
• InsufficiencyBench is introduced as a new legal benchmark targeting this issue.
• InsufficiencyBench assesses if a model can recognize when a query lacks legally important information, identify what is missing, and avoid premature conclusions.
• A taxonomy of eight canonical missing-element categories across three structural failure modes is provided.
• 202 benchmark items spanning six legal domains, 24 US jurisdictions, and annotated by practising attorneys are created.
• Ten frontier models are evaluated in the study.
• Results show that no model exceeds F2 = 0.46 on missing-element identification.
• Median recall is 0.44 for the models.

Key Concepts Mentioned

• Legal AI systems
• InsufficiencyBench
• query-side insufficiency
• taxonomy of missing-element categories
• frontier models

Claims Made by the Document

• Current legal AI systems struggle with query-side insufficiency.
• No model exceeds F2 = 0.46 on missing-element identification.

Evidence Used

• The study evaluated ten frontier models and found their performance to be suboptimal.
• The results show that the median recall is 0.44.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Legal AI systems are increasingly used to answer legal questions, yet existing benchmarks assume queries arrive fully specified. In practice, users omit facts that materially determine the legal outcome. We introduce InsufficiencyBench, the first legal benchmark targeting query-side insufficiency: whether a model recognizes when a query lacks legally material information, identifies what is missing, and refrains from premature conclusions. We formalize a taxonomy of eight canonical missing-element categories across three structural failure modes---switch, gating, and fatal prerequisite--- and construct 202 benchmark items (58 base queries, 144 deficient variants) spanning six legal domains and 24 US jurisdictions and annotated by practising attorneys. Evaluating ten frontier models, we find that no model exceeds F2 = 0.46 on missing-element identification and that the median recall is 0.44. Models either hedge indiscriminately or answer silently under fabricated presumptions. No model both identifies and qualifies responses to deficient queries while directly addressing complete ones.