Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

A Multi-Agent System for Autonomous, Fine-Tuning-Free Clinical Symptom Detection: Development and Validation Study

Source

arXiv

arxiv.org/abs/2607.12886v1

Read the source

Source

Plain-Language Brief

A new multi-agent system named Pythia was developed for automatically and independently creating and optimizing clinical concept extraction prompts without manual engineering or fine-tuning. The study compared Pythia with a curated lexicon across 72 signs and symptoms from 400 clinical notes representing 387 patients. Pythia showed improved specificity over the lexicon, especially for concepts where the lexicon labeled every note positive. However, sensitivity transfer weakened below 5% prevalence.

What the Document Says

  • Pythia is a multi-agent system that autonomously creates and optimizes clinical concept extraction prompts without manual engineering or fine-tuning.
  • The study compares Pythia with a curated lexicon across 72 signs and symptoms from 400 clinical notes representing 387 patients.
  • Pythia achieved mean sensitivity of 0.76 and specificity of 0.95, compared with 0.82 and 0.76 for the lexicon.
  • For 20 out of 62 directly comparable concepts, Pythia matched or exceeded the lexicon on both sensitivity and specificity.
  • For 14 concepts where the lexicon labeled every note positive, Pythia recovered mean specificity of 0.97 by requiring a present-tense, patient-attributed finding rather than any textual mention of a term.
  • Specificity transferred from development to validation with minimal degradation across prevalences, whereas sensitivity transfer weakened below 5% prevalence.
  • A BERT classifier fine-tuned per concept on the same development set achieved mean sensitivity of 0.23 and collapsed to zero sensitivity for concepts below roughly 5% prevalence.

Key Concepts Mentioned

  • Pythia
  • clinical notes
  • symptom detection
  • multi-agent system
  • autonomous
  • prompt optimization
  • extraction prompts

Claims Made by the Document

  • Pythia produces symptom extraction prompts that generalize effectively from development to validation while remaining deployable on local infrastructure.
  • Autonomous, fine-tuning-free prompt optimization can be effective for symptom extraction.

Evidence Used

  • achieved mean sensitivity of 0.76 and specificity of 0.95
  • comparison with a curated lexicon
  • specificity transfer with minimal degradation
  • sensitivity transfer weakened below 5% prevalence

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Clinical notes contain many of the signs and symptoms that bring patients to care, yet this information rarely reaches structured fields. Existing extraction approaches either rely on context-insensitive rules that generate false positives or on supervised models that require substantial fine-tuning. We present Pythia, a multi-agent system that autonomously writes and optimizes extraction prompts for clinical concepts without manual prompt engineering or fine-tuning. Running on a locally hosted open-weights model, Pythia keeps clinical notes on local infrastructure and selects prompts using development-set sensitivity and specificity. We compared Pythia with a curated lexicon across 72 signs and symptoms from 400 clinical notes representing 387 patients. Development (n=300) and validation (n=100) sets were partitioned independently for each concept. Pythia achieved mean sensitivity of 0.76 and specificity of 0.95, compared with 0.82 and 0.76 for the lexicon, and matched or exceeded the lexicon on both metrics for 20 of 62 directly comparable concepts. For 14 concepts where the lexicon labeled every note positive, Pythia recovered mean specificity of 0.97 by requiring a present-tense, patient-attributed finding rather than any textual mention of a term. Specificity transferred from development to validation with minimal degradation across prevalences, whereas sensitivity transfer weakened below 5% prevalence, reaching a mean gap of 0.25 below 2% prevalence. A BERT classifier fine-tuned per concept on the same development set achieved mean sensitivity of 0.23 and collapsed to zero sensitivity for concepts below roughly 5% prevalence. These findings suggest that autonomous, fine-tuning-free prompt optimization can produce symptom extraction prompts that generalize effectively from development to validation while remaining deployable on local infrastructure.