Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

A Multi-Agent System for Autonomous, Fine-Tuning-Free Clinical Symptom Detection: Development and Validation Study

Source

arXiv

arxiv.org/abs/2607.12886v1

Read the source

Source

Plain-Language Brief

A new system called Pythia was developed, which automatically generates and optimizes extraction prompts for clinical concepts without the need for manual prompt engineering or fine-tuning. The researchers compared this system with a curated lexicon and found that it achieved similar or better results in sensitivity and specificity for most of the symptoms studied.

What the Document Says

  • A multi-agent system called Pythia was presented, which autonomously writes and optimizes extraction prompts for clinical concepts without manual prompt engineering or fine-tuning.
  • Pythia runs on a locally hosted open-weights model and selects prompts using development-set sensitivity and specificity.
  • The researchers compared Pythia with a curated lexicon across 72 signs and symptoms from 400 clinical notes representing 387 patients.
  • Pythia achieved mean sensitivity of 0.76 and specificity of 0.95, compared to 0.82 and 0.76 for the lexicon.
  • For 20 out of 62 directly comparable concepts, Pythia matched or exceeded the lexicon on both sensitivity and specificity.
  • For 14 concepts where the lexicon labeled every note positive, Pythia recovered mean specificity of 0.97 by requiring a present-tense, patient-attributed finding rather than any textual mention of a term.
  • Specificity transferred from development to validation with minimal degradation across prevalences, whereas sensitivity transfer weakened below 5% prevalence.
  • A BERT classifier fine-tuned per concept on the same development set achieved mean sensitivity of 0.23 and collapsed to zero sensitivity for concepts below roughly 5% prevalence.

Key Concepts Mentioned

  • Pythia
  • multi-agent system
  • clinical notes
  • symptom extraction
  • prompts
  • development-set
  • sensitivity
  • specificity

Claims Made by the Document

  • Pythia can produce symptom extraction prompts that generalize effectively from development to validation.
  • Autonomous, fine-tuning-free prompt optimization can be effective for symptom extraction.

Evidence Used

  • Comparison of Pythia with a curated lexicon and the results achieved in sensitivity and specificity.
  • Results showing that Pythia matched or exceeded the lexicon on both metrics for 20 out of 62 directly comparable concepts.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Clinical notes contain many of the signs and symptoms that bring patients to care, yet this information rarely reaches structured fields. Existing extraction approaches either rely on context-insensitive rules that generate false positives or on supervised models that require substantial fine-tuning. We present Pythia, a multi-agent system that autonomously writes and optimizes extraction prompts for clinical concepts without manual prompt engineering or fine-tuning. Running on a locally hosted open-weights model, Pythia keeps clinical notes on local infrastructure and selects prompts using development-set sensitivity and specificity. We compared Pythia with a curated lexicon across 72 signs and symptoms from 400 clinical notes representing 387 patients. Development (n=300) and validation (n=100) sets were partitioned independently for each concept. Pythia achieved mean sensitivity of 0.76 and specificity of 0.95, compared with 0.82 and 0.76 for the lexicon, and matched or exceeded the lexicon on both metrics for 20 of 62 directly comparable concepts. For 14 concepts where the lexicon labeled every note positive, Pythia recovered mean specificity of 0.97 by requiring a present-tense, patient-attributed finding rather than any textual mention of a term. Specificity transferred from development to validation with minimal degradation across prevalences, whereas sensitivity transfer weakened below 5% prevalence, reaching a mean gap of 0.25 below 2% prevalence. A BERT classifier fine-tuned per concept on the same development set achieved mean sensitivity of 0.23 and collapsed to zero sensitivity for concepts below roughly 5% prevalence. These findings suggest that autonomous, fine-tuning-free prompt optimization can produce symptom extraction prompts that generalize effectively from development to validation while remaining deployable on local infrastructure.