Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Closed-Loop Validation-Repair for Healthcare Interoperability: A Multi-Model Study of Schema Compliance in Clinical LLMs

Source

arXiv

arxiv.org/abs/2607.24371v1

Read the source

Source

Plain-Language Brief

A study evaluates three large language models (Qwen2.5 7B, Llama 3.1 8B, and Gemma2 9B) on their compliance with healthcare standards such as ICD-10, CPT, and HL7 FHIR in 320 clinical scenarios. The study found that all models had a consistent level of schema noncompliance (85.9-91.6%), mostly due to using alternative medical abbreviations and code prefixes instead of following healthcare IT standards. However, the validation-repair framework improved compliance rates significantly, achieving 99.0% overall with most errors being corrected in one or two iterations.

What the Document Says

  • The study evaluates three large language models (Qwen2.5 7B, Llama 3.1 8B, and Gemma2 9B) on their compliance with healthcare standards such as ICD-10, CPT, and HL7 FHIR.
  • The evaluation is done across 320 clinical scenarios spanning ten medical specialties, yielding 960 model-scenario pairs assessed under paired baseline and validation-repair conditions.
  • Schema noncompliance is consistent across the three models, with baseline compliance rates ranging from 85.9 to 91.6%.
  • 96% of validator-detected failures are representation-level format violations such as alternative medical abbreviations and code prefixes.
  • The validation-repair framework achieves 99.0% overall compliance, ranging from 98.4 to 99.4% across models.
  • Most errors resolve within one or two iterations.
  • Exact McNemar p-values below 0.001 and absolute improvements of 7.8 to 12.5 percentage points across model sizes confirm statistical significance.

Key Concepts Mentioned

  • Healthcare interoperability
  • Large language models
  • Schema compliance
  • ICD-10
  • CPT
  • HL7 FHIR
  • Validation-repair

Claims Made by the Document

  • All three models have a consistent level of schema noncompliance.
  • The validation-repair framework significantly improves schema compliance rates.

Evidence Used

  • Baseline compliance rates ranging from 85.9 to 91.6%
  • Improvements of 7.8 to 12.5 percentage points across model sizes
  • Exact McNemar p-values below 0.001

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Healthcare interoperability requires AI systems to produce structured outputs conforming to standardized schemas including ICD-10 for diagnostic coding, CPT for procedure billing, and HL7 FHIR for data exchange. While large language models demonstrate clinical reasoning capabilities, their integration into electronic health record systems faces a critical barrier: schema noncompliance. We evaluate three open-source models, Qwen2.5 7B, Llama 3.1 8B, and Gemma2 9B, via local deployment across 320 clinical scenarios spanning ten medical specialties, yielding 960 model-scenario pairs assessed under paired baseline and validation-repair conditions. First, schema noncompliance is consistent across the three model families, with baseline compliance rates ranging from 85.9 to 91.6 percent despite varying architectures and training data, suggesting shared gaps in medical training corpora rather than model-specific limitations. Second, 96 percent of validator-detected failures are representation-level format violations such as alternative medical abbreviations and code prefixes, indicating models follow clinical writing conventions but lack awareness of healthcare IT standards. Third, the validation-repair framework achieves 99.0 percent overall compliance, ranging from 98.4 to 99.4 percent across models, with most errors resolving within one or two iterations. Exact McNemar p-values below 0.001 and absolute improvements of 7.8 to 12.5 percentage points across model sizes confirm statistical significance. These results support closed-loop validation-repair as an effective system-level safeguard for healthcare interoperability, improving schema-level readiness for downstream clinical system integration.