Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning

Source

arXiv

arxiv.org/abs/2606.23301v1

Read the source

Source

Plain-Language Brief

Researchers introduce EHR-Complex, a new benchmark for evaluating the performance of medical AI agents in handling complex clinical database reasoning tasks. The benchmark is built on MIMIC-IV data and comprises about 52,000 tasks spanning six clinical intents. It requires agents to interact with a sandboxed environment by executing SQL queries or Python code. The research shows that even the top-performing model achieves only 62.3% exact-match accuracy, and most models struggle with consistency and encounter common failure modes like SQL logic errors, medical-code lookup failures, and semantic misunderstandings.

What the Document Says

  • EHR-Complex is a new benchmark for clinical database reasoning.
  • It's built on MIMIC-IV data and has about 52,000 tasks.
  • Tasks span six clinical intents and support patient-level and population-level queries.
  • Agents must interact with a sandboxed environment using SQL queries or Python code.
  • EHR-Complex considers real-world SQL task complexity for longitudinal multi-table aggregation and compositional reasoning.
  • On average, each query has 31.93 SQL structural components.
  • Evaluation results show that even the top-performing model achieves only 62.3% exact-match accuracy.
  • Pass^k consistency drops below 50% for nearly all models at k=4.
  • Failure analysis of more than 3,800 failed trajectories reveals three dominant failure modes: SQL logic errors, medical-code lookup failures, and semantic misunderstandings.

Key Concepts Mentioned

  • EHR-Complex
  • MIMIC-IV
  • clinical intents
  • SQL queries
  • Python code
  • Longitudinal multi-table aggregation
  • compositional reasoning

Claims Made by the Document

  • Existing benchmarks fail to reflect the complexity of practical EHR analysis.
  • EHR-Complex provides a rigorous testbed for clinical agents.

Evidence Used

  • Clinical agents promise to democratize access to electronic health records (EHRs), yet existing benchmarks fail to reflect the complexity of practical EHR analysis
  • EHR-Complex provides a rigorous testbed for clinical agents and highlights remaining gaps in robust reasoning for large-scale EHR analysis.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Clinical agents promise to democratize access to electronic health records (EHRs), yet existing benchmarks fail to reflect the complexity of practical EHR analysis, e.g., often operating on idealized, clean EHRs via static SQL generation rather than interactive execution. In this work, we introduce EHR-Complex, a large-scale benchmark designed for interactive clinical database reasoning. Built on the large MIMIC-IV substrate (365K patients, 31 tables, 500M+ records), EHR-Complex comprises about 52K tasks spanning six clinical intents, supporting both patient-level and population-level queries, where each task requires an agent to interact with a sandboxed environment by executing SQL queries or Python code. Notably, EHR-Complex considers the real-world SQL task complexity for longitudinal multi-table aggregation and compositional reasoning, resulting in 31.93 SQL structural components per query on average. Evaluation results on EHR-Complex reveal the clinical difficulty of these EHR reasoning scenarios, with the top-performing model achieving only 62.3% exact-match accuracy. Pass^k consistency drops below 50% for nearly all evaluated models at k=4, exposing broad stochastic fragility. A fine-grained analysis of more than 3,800 failed trajectories for representative LLMs reveals three dominant failure modes: SQL logic errors, medical-code lookup failures, and semantic misunderstandings. EHR-Complex provides a rigorous testbed for clinical agents and highlights remaining gaps in robust reasoning for large-scale EHR analysis.