Improving the Realism of Synthetic Clinical Benchmarks Under Utility Constraints

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Omid Bazgir, Md Nasir, Jacob Hoffman, Yang Yang, Manu Agrawal, Anusua Trivedi, Vinay Rao Dandin, Chris Gibbons, Christine Swisher
• Venue: arXiv: cs.AI
• Published: 2026-08-06
• Collected: 2026-08-07
• URL: http://arxiv.org/abs/2608.06265v1
• PDF: https://arxiv.org/pdf/2608.06265v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers propose a method to improve the realism of synthetic clinical benchmarks for AI agents in privacy-sensitive healthcare settings without compromising their practical utility. They formulate this as a problem of improving realism under utility constraints and demonstrate this on a care-gap benchmark derived from Synthea-generated patients.

What the Document Says

• The study discusses synthetic clinical benchmarks for enterprise AI agents in healthcare settings.
• These benchmarks can be unrealistic, especially in privacy-sensitive scenarios where operational data is hard to access.
• The authors aim to enhance such benchmarks while maintaining their downstream utility checks already used in practice.
• They formulate the problem as a utility-constrained realism improvement: dataset changes should increase realism and maintain an operational utility floor.
• They work on a care-gap benchmark derived from Synthea-generated patients.
• Realism is measured through missingness structure, simplicity, structural plausibility, and population alignment.
• The baseline benchmark is found to be thin: sampled-pair missingness is 79.44%, only 12.75% of rows are actionable, 38.94% of patients have zero actionable measures, and top-three token concentration reaches 100.0%.
• Two deterministic revisions improve these panels while remaining above the current utility floor.

Key Concepts Mentioned

• synthetic clinical benchmarks
• enterprise AI agents
• privacy-sensitive healthcare settings
• utility-constrained realism improvement
• downstream utility checks
• care-gap benchmark
• Synthea-generated patients
• missingness structure
• simplicity
• structural plausibility
• population alignment

Claims Made by the Document

• Existing synthetic clinical benchmarks can be unrealistic.
• The authors propose a method to improve such benchmarks while maintaining their downstream utility checks.
• The proposed method formulates the problem as a utility-constrained realism improvement.

Evidence Used

• The study discusses the unrealistic nature of existing synthetic clinical benchmarks.
• The authors explain their approach to improve these benchmarks while maintaining their downstream utility checks.
• They formulate this as a problem of improving realism under utility constraints.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Synthetic clinical benchmarks for enterprise AI agents can pass existing utility checks and still remain structurally unrealistic, especially in privacy-sensitive healthcare settings where operational data are hard to access. We study how to improve such benchmarks without breaking the downstream utility checks already used in practice. We formulate benchmark revision as utility-constrained realism improvement: dataset changes should increase realism while staying above an operational utility floor. We instantiate this idea on a care-gap benchmark derived from Synthea-generated patients exercised through demonstration electronic health record workflows and then processed by the same downstream pipeline as operational data. Realism is measured through missingness structure, simplicity, structural plausibility, and population alignment. The baseline benchmark is extremely thin: sampled-pair missingness is 79.44%, only 12.75% of rows are actionable, 38.94% of patients have zero actionable measures, and top-three token concentration reaches 100.0%. Two deterministic revisions improve these panels while remaining above the current utility floor, whereas a naive densification control preserves unrealistic templating. We further show that internal benchmark realism and source fidelity to an aggregate operational reference are related but distinct objectives. These results suggest that synthetic benchmark quality should be optimized explicitly, with utility treated as one constraint rather than as sufficient evidence of realism.