Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agents

Source

arXiv

arxiv.org/abs/2607.18999v1

Read the source

Source

Plain-Language Brief

Researchers have developed a new testbed called MedDDC-Eval for evaluating multi-turn medical consultation agents. This testbed separates the quality of the elicited history from the terminal diagnosis generation, allowing for a more accurate assessment of an agent's performance. The study demonstrates that the testbed improves the performance of a policy by training it using diagnosis-result and trajectory feedback, showing its potential for controlled attribution, interpretable elicited-history measurement, and evaluation-guided evidence-acquisition policy development.

What the Document Says

  • The text discusses a new testbed called MedDDC-Eval designed for evaluating multi-turn medical consultation agents.
  • MedDDC-Eval separates the quality of the elicited history from the terminal diagnosis generation to provide more accurate assessments.
  • The study shows that using diagnosis-result and trajectory feedback can improve the performance of a policy.
  • The testbed is said to support controlled attribution, interpretable elicited-history measurement, and evaluation-guided evidence-acquisition policy development.

Key Concepts Mentioned

  • MedDDC-Eval
  • multi-turn medical consultation agents
  • policy-elicited history
  • terminal diagnosis generation
  • diagnosis F1
  • GRPO
  • Qwen3-32B

Claims Made by the Document

  • MedDDC-Eval provides a diagnosis-decoupled testbed for multi-turn medical consultation agents.
  • Changing the diagnostic reader in MedDDC-Eval shifts diagnosis F1 by 2.2-19.0 points and reverses 18% and 36% of policy orderings.

Evidence Used

  • The text states that MedDDC-Eval treats elicited history as the comparison object and holds the history-to-diagnosis mapping constant through a shared frozen reader.
  • The study shows that post-training Qwen3-32B using diagnosis-result and trajectory feedback improves performance.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Multi-turn medical consultation agents must decide what to ask, adapt to patient responses, and determine when the collected evidence is sufficient. However, coupled evaluation conflates the quality of the policy-elicited history with policy-specific terminal diagnosis generation: strong generation can compensate for a thin history, while weaker generation can obscure a rich one. We introduce MedDDC-Eval, a diagnosis-decoupled testbed that treats elicited history as the comparison object and holds the history-to-diagnosis mapping constant through a shared frozen reader. Across two held-out sources, a grounded interface and an auditable diagnosis-trajectory-efficiency (D/T/E) harness measure diagnostic usefulness, information acquisition, and efficiency. Directional semantic coverage followed by deterministic one-to-one assignment yields coherent precision-recall counts for open-ended items, with at most one credited match per prediction or reference. Holding histories fixed, changing only the diagnostic reader shifts diagnosis F1 by 2.2-19.0 points and reverses 18% and 36% of pairwise policy orderings on the Record and Dialogue splits. We further apply standard Group Relative Policy Optimization (GRPO) over interactive multi-turn rollouts to post-train Qwen3-32B using diagnosis-result and trajectory feedback. On the 100-case Record and 70-case Dialogue splits, the trained policy improves over its initialization by 9.7 and 4.6 total-score points; removing either primary signal lowers held-out joint performance. These results show that MedDDC-Eval supports controlled attribution, interpretable elicited-history measurement, and evaluation-guided evidence-acquisition policy development.