Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark

Source facts

The preprint describes a fully synthetic, fact-grounded longitudinal EHR benchmark built from public medical education material, with no PHI. The abstract reports 1,268 patients and 5,602 encounters, ICD-10-CM/SNOMED CT/LOINC ontologies, provenance, interoperability APIs, role-based access, and function calling. In blinded physician review, physicians were near chance at distinguishing records from real charts. Across ten frontier and open models, the best longitudinal problem-list severity-weighted F1 was 0.73, matching the physician mean on a matched subset; the abstract also reports that models missed roughly half of clinically relevant findings in chart summarization.

Analysis

This provides a potentially reproducible testbed for longitudinal clinical reasoning while keeping the data synthetic. Its central signal is not a product claim but a benchmark boundary: even when aggregate scores match a physician mean on one subset, clinically relevant omissions remain material.

Uncertainty/Doubt

• This is an arXiv preprint. The physician validation protocol, synthetic-to-real transfer, model configurations, and clinical significance of the reported scores were not independently audited here.