Improving the Realism of Synthetic Clinical Benchmarks Under Utility Constraints

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Omid Bazgir, Md Nasir, Jacob Hoffman, Yang Yang, Manu Agrawal, Anusua Trivedi, Vinay Rao Dandin, Chris Gibbons, Christine Swisher
• Venue: arXiv: cs.AI
• Published: 2026-08-06
• Collected: 2026-08-07
• URL: http://arxiv.org/abs/2608.06265v1
• PDF: https://arxiv.org/pdf/2608.06265v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers propose a method to create more realistic synthetic clinical benchmarks for AI agents in healthcare settings, where operational data is hard to access. The method aims to improve the benchmark's realism while maintaining its utility, addressing issues like high missingness and low actionability. The study compares two deterministic revisions with a naive densification control, finding that the proposed methods improve realism without compromising utility.

What the Document Says

• The paper discusses improving synthetic clinical benchmarks for AI agents in healthcare settings
• These benchmarks may be structurally unrealistic despite passing existing utility checks
• The authors propose a method to improve these benchmarks while maintaining their utility
• They call this approach 'utility-constrained realism improvement'
• The study is based on a care-gap benchmark derived from Synthea-generated patients
• Realism is measured through missingness structure, simplicity, structural plausibility, and population alignment
• The baseline benchmark has high missingness (79.44%), low actionability (12.75%), and high token concentration (100.0%)
• Two deterministic revisions improve these panels while remaining above the current utility floor
• A naive densification control preserves unrealistic templating
• The study finds that internal benchmark realism and source fidelity to an aggregate operational reference are related but distinct objectives

Key Concepts Mentioned

• synthetic clinical benchmarks
• AI agents
• healthcare settings
• utility-constrained realism improvement
• missingness structure
• simplicity
• structural plausibility
• population alignment
• care-gap benchmark
• Synthea-generated patients
• electronic health record workflows
• downstream pipeline
• operational data
• utility floor
• deterministic revisions
• naive densification control
• token concentration
• internal benchmark realism
• source fidelity

Claims Made by the Document

• Synthetic clinical benchmarks can be structurally unrealistic despite passing utility checks
• The proposed method, 'utility-constrained realism improvement', improves the realism of synthetic clinical benchmarks while maintaining their utility
• Two deterministic revisions improve the realism of panels without compromising utility
• A naive densification control preserves unrealistic templating
• Internal benchmark realism and source fidelity to an aggregate operational reference are related but distinct objectives

Evidence Used

• The text describes a study and its findings on improving synthetic clinical benchmarks for AI agents

Methods / Approach

• Formulating benchmark revision as utility-constrained realism improvement
• Using a care-gap benchmark derived from Synthea-generated patients
• Measuring realism through missingness structure, simplicity, structural plausibility, and population alignment

Limitations Stated by the Authors

• The paper does not discuss the potential challenges or limitations of implementing the proposed method in practice

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The study does not provide specific details on how the 'utility floor' is determined
• It is unclear if the findings can be generalized to other types of AI applications outside healthcare

Open Questions

• What are the potential challenges or limitations of implementing the proposed method in practice?
• Can the findings be generalized to other types of AI applications outside healthcare?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The paper does not discuss the potential challenges or limitations of implementing the proposed method in practice. This could include issues related to data availability, computational resources, and the generalizability of the results.
• The study's findings may not be directly applicable to other types of AI applications outside healthcare. It would be interesting to explore if similar methods could be applied to improve realism in those contexts.

Raw Abstract / Excerpt

Synthetic clinical benchmarks for enterprise AI agents can pass existing utility checks and still remain structurally unrealistic, especially in privacy-sensitive healthcare settings where operational data are hard to access. We study how to improve such benchmarks without breaking the downstream utility checks already used in practice. We formulate benchmark revision as utility-constrained realism improvement: dataset changes should increase realism while staying above an operational utility floor. We instantiate this idea on a care-gap benchmark derived from Synthea-generated patients exercised through demonstration electronic health record workflows and then processed by the same downstream pipeline as operational data. Realism is measured through missingness structure, simplicity, structural plausibility, and population alignment. The baseline benchmark is extremely thin: sampled-pair missingness is 79.44%, only 12.75% of rows are actionable, 38.94% of patients have zero actionable measures, and top-three token concentration reaches 100.0%. Two deterministic revisions improve these panels while remaining above the current utility floor, whereas a naive densification control preserves unrealistic templating. We further show that internal benchmark realism and source fidelity to an aggregate operational reference are related but distinct objectives. These results suggest that synthetic benchmark quality should be optimized explicitly, with utility treated as one constraint rather than as sufficient evidence of realism.