Improving the Realism of Synthetic Clinical Benchmarks Under Utility Constraints

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Omid Bazgir, Md Nasir, Jacob Hoffman, Yang Yang, Manu Agrawal, Anusua Trivedi, Vinay Rao Dandin, Chris Gibbons, Christine Swisher
• Venue: arXiv: cs.AI
• Published: 2026-08-06
• Collected: 2026-08-07
• URL: http://arxiv.org/abs/2608.06265v1
• PDF: https://arxiv.org/pdf/2608.06265v1
• DOI: UNKNOWN

Plain-Language Brief

The authors propose a method to improve the realism of synthetic clinical benchmarks for enterprise AI agents while maintaining their utility. They focus on healthcare settings where operational data is hard to access and create a care-gap benchmark using Synthea-generated patients. The benchmark's realism is measured through missingness structure, simplicity, structural plausibility, and population alignment. The results show that two deterministic revisions improve the benchmark while maintaining utility, whereas a naive densification control does not.

What the Document Says

• The authors discuss improving synthetic clinical benchmarks for enterprise AI agents in privacy-sensitive healthcare settings.
• They formulate benchmark revision as a problem of increasing realism while staying above an operational utility floor.
• The study uses a care-gap benchmark derived from Synthea-generated patients and processes it through the same downstream pipeline as operational data.
• Realism is measured through missingness structure, simplicity, structural plausibility, and population alignment.
• The baseline benchmark has high levels of missing data, low actionability, and high token concentration.
• Two deterministic revisions improve the benchmark while maintaining utility, whereas a naive densification control does not.
• Internal benchmark realism and source fidelity to an aggregate operational reference are related but distinct objectives.

Key Concepts Mentioned

• synthetic clinical benchmarks
• enterprise AI agents
• privacy-sensitive healthcare settings
• utility-constrained realism improvement
• care-gap benchmark
• Synthea-generated patients
• electronic health record workflows
• missingness structure
• simplicity
• structural plausibility
• population alignment

Claims Made by the Document

• Structurally unrealistic synthetic clinical benchmarks can pass existing utility checks.
• Deterministic revisions improve the realism of synthetic clinical benchmarks while maintaining utility.
• Naive densification control does not improve the realism of synthetic clinical benchmarks.

Evidence Used

• The text describes a study and its findings regarding synthetic clinical benchmarks.
• Data from the care-gap benchmark is analyzed and compared to operational data.

Methods / Approach

• Formulation of benchmark revision as utility-constrained realism improvement
• Creation of a care-gap benchmark using Synthea-generated patients
• Processing the benchmark through electronic health record workflows
• Measuring realism through missingness structure, simplicity, structural plausibility, and population alignment

Limitations Stated by the Authors

Not stated in source

Limitations Noted by the Agent

Agent interpretation — not from the source.

The study only examines one type of synthetic clinical benchmark and does not explore other types or settings.

Open Questions

• What are the potential implications of these findings for the development of enterprise AI agents?
• How can this method be applied to other types of synthetic clinical benchmarks?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The authors suggest that synthetic benchmark quality should be optimized explicitly, with utility treated as one constraint rather than sufficient evidence of realism.
• It is unclear whether the improvement in realism leads to improved performance of enterprise AI agents.

Raw Abstract / Excerpt

Synthetic clinical benchmarks for enterprise AI agents can pass existing utility checks and still remain structurally unrealistic, especially in privacy-sensitive healthcare settings where operational data are hard to access. We study how to improve such benchmarks without breaking the downstream utility checks already used in practice. We formulate benchmark revision as utility-constrained realism improvement: dataset changes should increase realism while staying above an operational utility floor. We instantiate this idea on a care-gap benchmark derived from Synthea-generated patients exercised through demonstration electronic health record workflows and then processed by the same downstream pipeline as operational data. Realism is measured through missingness structure, simplicity, structural plausibility, and population alignment. The baseline benchmark is extremely thin: sampled-pair missingness is 79.44%, only 12.75% of rows are actionable, 38.94% of patients have zero actionable measures, and top-three token concentration reaches 100.0%. Two deterministic revisions improve these panels while remaining above the current utility floor, whereas a naive densification control preserves unrealistic templating. We further show that internal benchmark realism and source fidelity to an aggregate operational reference are related but distinct objectives. These results suggest that synthetic benchmark quality should be optimized explicitly, with utility treated as one constraint rather than as sufficient evidence of realism.