Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning

Source

arXiv

arxiv.org/abs/2606.23301v1

Read the source

Source

Plain-Language Brief

Researchers have introduced a new benchmark called EHR-Complex to test the performance of clinical agents that interact with electronic health records (EHRs). The benchmark is built on a large dataset from MIMIC-IV and includes about 52,000 tasks across six clinical intents. Each task requires an agent to execute SQL queries or Python code within a sandboxed environment. The results show that the top-performing model only achieved 62.3% exact-match accuracy and that most models struggle with consistency beyond k=4. A detailed analysis of the failures reveals three main issues: SQL logic errors, medical-code lookup failures, and semantic misunderstandings.

What the Document Says

  • The researchers introduce EHR-Complex, a large-scale benchmark for clinical database reasoning.
  • EHR-Complex is built on the MIMIC-IV dataset with about 52,000 tasks.
  • Tasks cover six clinical intents and support both patient-level and population-level queries.
  • Each task requires an agent to interact with a sandboxed environment via SQL queries or Python code.
  • EHR-Complex considers real-world SQL task complexity for longitudinal multi-table aggregation and compositional reasoning.
  • The average query has 31.93 SQL structural components.
  • Evaluation results show that the top-performing model achieves 62.3% exact-match accuracy.
  • Pass^k consistency drops below 50% for most models at k=4, indicating stochastic fragility.
  • A detailed analysis of more than 3,800 failed trajectories reveals three main failure modes: SQL logic errors, medical-code lookup failures, and semantic misunderstandings.

Key Concepts Mentioned

  • EHR-Complex
  • clinical agents
  • electronic health records (EHRs)
  • MIMIC-IV
  • SQL queries
  • Python code
  • patient-level and population-level queries
  • longitudinal multi-table aggregation
  • compositional reasoning
  • exact-match accuracy
  • Pass^k consistency

Claims Made by the Document

  • Existing benchmarks for clinical agents fail to reflect the complexity of practical EHR analysis.
  • EHR-Complex is designed to test the performance of clinical agents on complex EHR analysis tasks.
  • The top-performing model achieves only 62.3% exact-match accuracy on EHR-Complex.
  • Pass^k consistency drops below 50% for most models at k=4 on EHR-Complex.
  • A detailed analysis of more than 3,800 failed trajectories reveals three main failure modes.

Evidence Used

  • The text states that existing benchmarks fail to reflect the complexity of practical EHR analysis and introduces EHR-Complex as a solution.
  • The text describes the characteristics of EHR-Complex, including its construction on the MIMIC-IV dataset and the number of tasks it contains.
  • The text mentions that each task requires an agent to interact with a sandboxed environment via SQL queries or Python code.
  • The text states that the average query has 31.93 SQL structural components and discusses the real-world SQL task complexity in EHR-Complex.
  • The text presents the evaluation results on EHR-Complex, including the exact-match accuracy of the top-performing model and the Pass^k consistency for most models.
  • The text provides a detailed analysis of more than 3,800 failed trajectories in EHR-Complex and identifies three main failure modes.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Clinical agents promise to democratize access to electronic health records (EHRs), yet existing benchmarks fail to reflect the complexity of practical EHR analysis, e.g., often operating on idealized, clean EHRs via static SQL generation rather than interactive execution. In this work, we introduce EHR-Complex, a large-scale benchmark designed for interactive clinical database reasoning. Built on the large MIMIC-IV substrate (365K patients, 31 tables, 500M+ records), EHR-Complex comprises about 52K tasks spanning six clinical intents, supporting both patient-level and population-level queries, where each task requires an agent to interact with a sandboxed environment by executing SQL queries or Python code. Notably, EHR-Complex considers the real-world SQL task complexity for longitudinal multi-table aggregation and compositional reasoning, resulting in 31.93 SQL structural components per query on average. Evaluation results on EHR-Complex reveal the clinical difficulty of these EHR reasoning scenarios, with the top-performing model achieving only 62.3% exact-match accuracy. Pass^k consistency drops below 50% for nearly all evaluated models at k=4, exposing broad stochastic fragility. A fine-grained analysis of more than 3,800 failed trajectories for representative LLMs reveals three dominant failure modes: SQL logic errors, medical-code lookup failures, and semantic misunderstandings. EHR-Complex provides a rigorous testbed for clinical agents and highlights remaining gaps in robust reasoning for large-scale EHR analysis.

Links to this note 1