Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

A Multi-Agent System for Autonomous, Fine-Tuning-Free Clinical Symptom Detection: Development and Validation Study

Source

arXiv

arxiv.org/abs/2607.12886v1

Read the source

Source

Plain-Language Brief

Researchers present Pythia, a multi-agent system that automatically generates and optimizes extraction prompts for clinical concepts without manual engineering or fine-tuning. They compare Pythia with a curated lexicon across various symptoms in 400 clinical notes. Pythia performed better or equally well than the lexicon in terms of sensitivity and specificity for most concepts, especially when dealing with false positives. However, its performance weakened below 5% prevalence.

What the Document Says

  • The research presents a multi-agent system named Pythia.
  • Pythia is designed to autonomously write and optimize extraction prompts for clinical concepts without manual prompt engineering or fine-tuning.
  • It runs on a locally hosted open-weights model.
  • Pythia keeps clinical notes on local infrastructure and selects prompts using development-set sensitivity and specificity.
  • The study compares Pythia with a curated lexicon across 72 signs and symptoms from 400 clinical notes representing 387 patients.
  • Development (n=300) and validation (n=100) sets were partitioned independently for each concept.
  • Pythia achieved mean sensitivity of 0.76 and specificity of 0.95, compared with 0.82 and 0.76 for the lexicon.
  • For most concepts, Pythia matched or exceeded the lexicon on both metrics.
  • For 14 concepts where the lexicon labeled every note positive, Pythia recovered mean specificity of 0.97 by requiring a present-tense, patient-attributed finding rather than any textual mention of a term.
  • Specificity transferred from development to validation with minimal degradation across prevalences.
  • Sensitivity transfer weakened below 5% prevalence, reaching a mean gap of 0.25 below 2% prevalence.
  • A BERT classifier fine-tuned per concept on the same development set achieved mean sensitivity of 0.23 and collapsed to zero sensitivity for concepts below roughly 5% prevalence.

Key Concepts Mentioned

  • Pythia
  • multi-agent system
  • autonomous
  • extraction prompts
  • clinical concepts
  • local infrastructure
  • development set
  • validation set
  • sensitivity
  • specificity
  • BERT classifier

Claims Made by the Document

  • Pythia can produce symptom extraction prompts that generalize effectively from development to validation.
  • Pythia performs better or equally well than a curated lexicon in terms of sensitivity and specificity for most concepts.

Evidence Used

  • mean sensitivity of 0.76 and specificity of 0.95 (for Pythia)
  • mean sensitivity of 0.82 and specificity of 0.76 (for the lexicon)
  • matching or exceeding the lexicon on both metrics for 20 of 62 directly comparable concepts

Methods / Approach

  • comparison with a curated lexicon
  • use of locally hosted open-weights model
  • selection of prompts using development-set sensitivity and specificity

Limitations Stated by the Authors

Sensitivity transfer weakened below 5% prevalence.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The study does not discuss the computational resources required for Pythia's operation or its scalability.

Open Questions

  • What are the computational resources required for Pythia's operation?
  • How does Pythia handle ambiguous terms or contexts?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The study focuses on Pythia's performance compared to a lexicon and its ability to generalize from development to validation. It does not provide information about the exact nature of the curated lexicon, the patient population, or the specific clinical notes used.

Raw Abstract / Excerpt

Clinical notes contain many of the signs and symptoms that bring patients to care, yet this information rarely reaches structured fields. Existing extraction approaches either rely on context-insensitive rules that generate false positives or on supervised models that require substantial fine-tuning. We present Pythia, a multi-agent system that autonomously writes and optimizes extraction prompts for clinical concepts without manual prompt engineering or fine-tuning. Running on a locally hosted open-weights model, Pythia keeps clinical notes on local infrastructure and selects prompts using development-set sensitivity and specificity. We compared Pythia with a curated lexicon across 72 signs and symptoms from 400 clinical notes representing 387 patients. Development (n=300) and validation (n=100) sets were partitioned independently for each concept. Pythia achieved mean sensitivity of 0.76 and specificity of 0.95, compared with 0.82 and 0.76 for the lexicon, and matched or exceeded the lexicon on both metrics for 20 of 62 directly comparable concepts. For 14 concepts where the lexicon labeled every note positive, Pythia recovered mean specificity of 0.97 by requiring a present-tense, patient-attributed finding rather than any textual mention of a term. Specificity transferred from development to validation with minimal degradation across prevalences, whereas sensitivity transfer weakened below 5% prevalence, reaching a mean gap of 0.25 below 2% prevalence. A BERT classifier fine-tuned per concept on the same development set achieved mean sensitivity of 0.23 and collapsed to zero sensitivity for concepts below roughly 5% prevalence. These findings suggest that autonomous, fine-tuning-free prompt optimization can produce symptom extraction prompts that generalize effectively from development to validation while remaining deployable on local infrastructure.