Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Guideline-as-Oracle: Zero-Annotation Training of an Ophthalmic Telephone Triage Agent

Source

arXiv

arxiv.org/abs/2608.04772v1

Read the source

Source

Plain-Language Brief

Researchers present Guideline-as-Oracle (GAO), a method for training an ophthalmic telephone triage agent using guidelines from the American Academy of Ophthalmology as the only source of instance-level supervision during training. The method improves agreement with a reference and emergent-case recall when compared to general-purpose systems, and the gains persist across different datasets.

What the Document Says

  • The paper discusses Guideline-as-Oracle (GAO) for scaling supervision of multi-turn medical agents.
  • GAO uses guidelines from the American Academy of Ophthalmology to create a rule table with 70 rows for instance-level supervision during training.
  • The method trains on 3,000 dialogues, reserving human labeling for evaluation.
  • 8 construction strategies are cataloged for converting rules into dialogues.
  • GAO-Triage is the fine-tuned model resulting from this approach.
  • GAO-Triage shows improvement in agreement with a reference and emergent-case recall when compared to general-purpose systems.
  • The gains persist across a second seed and patient simulator.
  • None of the tested general-purpose systems dominates GAO-Triage on both metrics.
  • GAO-Triage requires no frontier model at inference time.
  • Permuting label-dialogue assignments causes the model to collapse into a constant-routine predictor, indicating that the signal lies in guideline-derived assignment rather than dialogue surface form.
  • Label repair coincides with the disappearance of a late-training safety degradation.

Key Concepts Mentioned

  • Guideline-as-Oracle (GAO)
  • American Academy of Ophthalmology guidelines
  • Multi-turn medical agents
  • Telephone triage agent
  • Rule table
  • Dialogue construction strategies
  • General-purpose systems
  • Frontier model

Claims Made by the Document

  • GAO improves agreement with a reference and emergent-case recall when compared to general-purpose systems.
  • The gains persist across different datasets.

Evidence Used

  • Improved metrics (agreement with a reference and emergent-case recall) are reported in the results section.

Methods / Approach

  • Fine-tuning a 9B backbone on the GAO corpus.
  • Using guidelines as instance-level supervision during training.

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Scaling supervision for multi-turn medical agents is difficult because expert dialogue annotation is costly and clinical conversations are privacy-restricted. We introduce Guideline-as-Oracle (GAO), which compiles American Academy of Ophthalmology guidance into a 70-row operational rule table and uses it as the sole source of instance-level supervision for 3,000 training dialogues, reserving human labeling for evaluation. Because converting rules into dialogues is itself a design problem, we catalog eight construction strategies, including cited-row tier assignment, one-fact boundary pairs, metadata-only repair, and label repair, and characterize the evidential status of each: labeling mechanism, null, confounded, or evaluated only as a package. Fine-tuning a 9B backbone on this corpus yields GAO-Triage, improving agreement with a 201-case operational reference from 61.7% to 74.1% (exact McNemar p=0.0046) and emergent-case recall from 9.5% to 69.0%; the gains persist across a second seed and patient simulator. None of the seven general-purpose systems we test dominates GAO-Triage on both metrics, and GAO-Triage requires no frontier model at inference time. Permuting label-dialogue assignments collapses the model to a constant-routine predictor, indicating that the signal lies in guideline-derived assignment rather than dialogue surface form. Label repair coincides with the disappearance of a late-training safety degradation.