Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs

Source

arXiv

arxiv.org/abs/2607.18086v1

Read the source

Source

Plain-Language Brief

A study investigates the impact of a structured evidence-sufficiency prompt on reducing overconfident answers from four language models in a clinical context. The study finds a significant reduction in unsafe overconfidence but notes that the extent of this reduction varies among judges, and it comes at a cost to model helpfulness.

What the Document Says

  • The study used four language models (GPT-5.5, Claude Opus 4.8, Gemini 3.5 Flash, Grok 4.3) and a structured evidence-sufficiency prompt to examine the reduction in unsafe overconfident answers.
  • The primary judge (GPT-5.4-nano) recorded a decrease in unsafe overconfidence from 49.3% to 24.7%.
  • The effect was found to be judge-dependent, with a secondary judge (Claude Sonnet 5) nearly halving the reduction effect.
  • Blinded clinicians characterized the primary judge as having high sensitivity but low specificity, indicating potential calibration issues.
  • The study also noted a model-specific helpfulness cost associated with the gain in safety.
  • Matched scaffold controls were used to confirm genuine behavior change, not judge circularity.
  • The conclusions suggest that clinical safety effects should be reported relative and anchored to human review.

Key Concepts Mentioned

  • language models
  • safety
  • overconfidence
  • evidence-sufficiency prompt
  • judge
  • clinical context

Claims Made by the Document

  • Using a structured evidence-sufficiency prompt reduces unsafe overconfidence in language models.
  • The effect of reducing unsafe overconfidence is judge-dependent.
  • There is a model-specific helpfulness cost associated with the gain in safety.

Evidence Used

  • Unsafe overconfidence fell from 49.3% to 24.7%
  • Magnitude was judge-dependent: Sonnet agreed on direction but nearly halved the effect +13.1 points

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Background: LLM judges increasingly score whether clinical language models give overconfident answers under incomplete evidence, yet whether a measured 'safety gain' reflects real behavior change or the judge's calibration is unresolved. Using a structured evidence-sufficiency prompt as a test case, we asked whether it reduces unsafe overconfident answers, how far that effect depends on the scoring judge, and what it costs in helpfulness. Methods: In a retrospective public-data benchmark (Real-POCQi, HealthBench, MedRBench), four models (GPT-5.5, Claude Opus 4.8, Gemini 3.5 Flash, Grok 4.3) answered a fully paired common panel (1,200 cells) with a standard prompt and the wrapper. The pre-specified endpoint was the paired reduction in unsafe overconfidence scored by the primary judge (GPT-5.4-nano); secondary analyses added a different-family judge (Claude Sonnet 5), a correctness judge, matched scaffold controls, and a blinded three-clinician review. Results: Unsafe overconfidence fell from 49.3% to 24.7%, a paired reduction of 24.7 points (95% CI 21.8-27.7; p<0.001), robust in direction across models and paraphrases. Magnitude was judge-dependent: Sonnet agreed on direction but nearly halved the effect (+13.1 points), with one-directional disagreement. Blinded clinicians characterized the primary judge as a high-sensitivity (1.00), low-specificity (0.55) screen, not a calibrated rate. The gain carried a model-specific helpfulness cost (correct diagnosis 80.3% to 50.3%): near-free for GPT-5.5, near-total for Gemini (-58 points). Matched scaffold controls showed genuine behavior change, not judge circularity. Conclusions: LLM-judged clinical safety effects should be reported as directional and relative, anchored to human review and evaluated jointly with helpfulness, not as calibrated absolute rates. This does not establish clinical deployment readiness.