Guideline-as-Oracle: Zero-Annotation Training of an Ophthalmic Telephone Triage Agent

Source

Source name: arXiv
Source kind: paper
Authors / organization: Chenyu Wang, Yi Liu, Baoqing Li, Min Tu, Diping Song
Venue: arXiv: cs.CL
Published: 2026-08-05
Collected: 2026-08-06
URL: http://arxiv.org/abs/2608.04772v1
PDF: https://arxiv.org/pdf/2608.04772v1
DOI: UNKNOWN

Plain-Language Brief

Researchers present Guideline-as-Oracle (GAO), a method to train an ophthalmic telephone triage agent without relying on expensive expert annotations or confidential clinical conversations. Instead, they use a rule table based on guidelines from the American Academy of Ophthalmology as the sole source of supervision for 3,000 training dialogues. The researchers developed eight construction strategies to convert these rules into dialogue formats and tested the resulting model (GAO-Triage) against a reference dataset. The new method improved agreement and emergent-case recall compared to seven general-purpose systems, demonstrating its potential in providing effective ophthalmic telephone triage.

What the Document Says

The study introduces Guideline-as-Oracle (GAO), a method that uses American Academy of Ophthalmology guidelines to train an ophthalmic telephone triage agent.
GAO uses a rule table with 70 rows as the sole source of supervision for 3,000 training dialogues.
Eight construction strategies were developed to convert rules into dialogue formats.
The study tests GAO-Triage against a reference dataset and compares it to seven general-purpose systems.
GAO-Triage shows improvements in agreement and emergent-case recall compared to the other systems.

Key Concepts Mentioned

Guideline-as-Oracle
ophthalmic telephone triage agent
American Academy of Ophthalmology guidelines
rule table
dialogue formats
training dialogues
evaluation
fine-tuning
9B backbone
operational reference
general-purpose systems

Claims Made by the Document

GAO improves agreement with a reference dataset.
GAO improves emergent-case recall compared to seven general-purpose systems.

Evidence Used

Improvement in exact McNemar p=0.0046 when comparing GAO-Triage's performance against a reference dataset.
Increase in emergent-case recall from 9.5% to 69.0% for GAO-Triage compared to the other systems.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Scaling supervision for multi-turn medical agents is difficult because expert dialogue annotation is costly and clinical conversations are privacy-restricted. We introduce Guideline-as-Oracle (GAO), which compiles American Academy of Ophthalmology guidance into a 70-row operational rule table and uses it as the sole source of instance-level supervision for 3,000 training dialogues, reserving human labeling for evaluation. Because converting rules into dialogues is itself a design problem, we catalog eight construction strategies, including cited-row tier assignment, one-fact boundary pairs, metadata-only repair, and label repair, and characterize the evidential status of each: labeling mechanism, null, confounded, or evaluated only as a package. Fine-tuning a 9B backbone on this corpus yields GAO-Triage, improving agreement with a 201-case operational reference from 61.7% to 74.1% (exact McNemar p=0.0046) and emergent-case recall from 9.5% to 69.0%; the gains persist across a second seed and patient simulator. None of the seven general-purpose systems we test dominates GAO-Triage on both metrics, and GAO-Triage requires no frontier model at inference time. Permuting label-dialogue assignments collapses the model to a constant-routine predictor, indicating that the signal lies in guideline-derived assignment rather than dialogue surface form. Label repair coincides with the disappearance of a late-training safety degradation.