Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Closed-Loop Validation-Repair for Healthcare Interoperability: A Multi-Model Study of Schema Compliance in Clinical LLMs

Source

arXiv

arxiv.org/abs/2607.24371v1

Read the source

Source

Plain-Language Brief

The study evaluates three large language models (Qwen2.5 7B, Llama 3.1 8B, and Gemma2 9B) in the context of healthcare interoperability. The models were assessed across 320 clinical scenarios for their compliance with standardized schemas like ICD-10, CPT, and HL7 FHIR. The study found that schema noncompliance is consistent across all three model families, but most failures are due to representation-level format violations. A validation-repair framework was introduced which improved overall compliance to 99.0%.

What the Document Says

  • The study examines three open-source models (Qwen2.5 7B, Llama 3.1 8B, and Gemma2 9B) in healthcare interoperability.
  • The study deploys these models across 320 clinical scenarios spanning ten medical specialties.
  • Baseline schema compliance rates range from 85.9 to 91.6% despite varying architectures and training data.
  • 96% of detected failures are representation-level format violations.
  • The validation-repair framework achieves overall compliance of 99.0% across models.
  • Exact McNemar p-values below 0.001 and absolute improvements of 7.8 to 12.5 percentage points confirm statistical significance.

Key Concepts Mentioned

  • Healthcare interoperability
  • AI systems
  • Structured outputs
  • Standardized schemas
  • ICD-10
  • CPT
  • HL7 FHIR
  • Large language models
  • Schema compliance
  • Validation-repair framework

Claims Made by the Document

  • Baseline schema compliance is inconsistent across the three model families.
  • Most detected failures are due to representation-level format violations.
  • The validation-repair framework significantly improves overall schema compliance.

Evidence Used

  • Compliance rates provided in text.
  • Percentage of format violations stated in text.
  • McNemar p-values and improvements mentioned in text.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Healthcare interoperability requires AI systems to produce structured outputs conforming to standardized schemas including ICD-10 for diagnostic coding, CPT for procedure billing, and HL7 FHIR for data exchange. While large language models demonstrate clinical reasoning capabilities, their integration into electronic health record systems faces a critical barrier: schema noncompliance. We evaluate three open-source models, Qwen2.5 7B, Llama 3.1 8B, and Gemma2 9B, via local deployment across 320 clinical scenarios spanning ten medical specialties, yielding 960 model-scenario pairs assessed under paired baseline and validation-repair conditions. First, schema noncompliance is consistent across the three model families, with baseline compliance rates ranging from 85.9 to 91.6 percent despite varying architectures and training data, suggesting shared gaps in medical training corpora rather than model-specific limitations. Second, 96 percent of validator-detected failures are representation-level format violations such as alternative medical abbreviations and code prefixes, indicating models follow clinical writing conventions but lack awareness of healthcare IT standards. Third, the validation-repair framework achieves 99.0 percent overall compliance, ranging from 98.4 to 99.4 percent across models, with most errors resolving within one or two iterations. Exact McNemar p-values below 0.001 and absolute improvements of 7.8 to 12.5 percentage points across model sizes confirm statistical significance. These results support closed-loop validation-repair as an effective system-level safeguard for healthcare interoperability, improving schema-level readiness for downstream clinical system integration.