Towards Expert-level Medical AI for Real-time Video Consultations

Source

Source name: arXiv
Source kind: paper
Authors / organization: Mahvish Nagda, Jihyeon Lee, Matthew Thompson, Chunjong Park, Tim Strother, Valentin Liévin, Roma Ruparel, Akshay Goel, Teya Bergamaschi, Suhana Bedi, Meet Shah, Pavel Dubov, Liviu Panait, Toshiyuki Fukuzawa, Sam Schmidgall, Craig Schiff, Joseph Xu, Aliya Rysbek, Yana Lunts, Jan Freyberg, Rebecca Hemengway, Sunny Virmani, David Racz, Carey Radebaugh, Joëlle Barral, Kavi Goel, Dale R. Webster, Katherine Chou, Avinatan Hassidim, Yossi Matias, James Manyika, Gregory Wayne, Tao Tu, Yun Liu, Ethan Goh, Christina Chen, Ryutaro Tanno, Po-Hsuan Cameron Chen, Mike Schaekermann, Anil Palepu
Venue: arXiv: cs.AI
Published: 2026-08-10
Collected: 2026-08-11
URL: http://arxiv.org/abs/2608.09861v1
PDF: https://arxiv.org/pdf/2608.09861v1
DOI: UNKNOWN
Registry group: academic_apis_and_preprints
Evidence role: canonical_academic_index
Language: en
Country / region: UNKNOWN

Plain-Language Brief

A new AI system called AMIE (Articulate Medical Intelligence Explorer) in a video configuration has been demonstrated to perform at expert-level for real-time clinical video consultations. This system, which integrates low-latency dialogue, clinical reasoning, and real-time audio-visual perception, was compared to human primary care physicians (PCPs) and its text-only counterpart in a study involving 30 PCPs, 15 patient actors, and 100 clinical scenarios. The AI system was rated on par or better than PCPs in history-taking, diagnosis, management, and physical observation and examination. Patient actors preferred the AI's approach to assessing and explaining conditions, while PCPs were preferred for rapport and partnership building.

What the Document Says

The study involves an AI system called AMIE (Video) which is a multi-agent system based on Gemini, integrating low-latency dialogue, clinical reasoning, and real-time audio-visual perception for real-time clinical video consultations.
A comparison was made between AMIE (Video), its text-only counterpart AMIE (Text), and human primary care physicians (PCPs) in a study with 30 PCPs, 15 patient actors, and 100 clinical scenarios.
Clinical evaluators rated AMIE (Video) on par or better than PCPs in history-taking, diagnosis, management, and physical observation and examination.
Patient actors preferred AMIE's approach to assessing and explaining conditions, while PCPs were preferred for rapport and partnership building.
In modality ablation, patient actors preferred AMIE (Video)'s interface over text chat for communicative effectiveness, convenience, and feeling understood.
Limitations remain in fine anatomical precision, subtle affective nuances, and high-frequency movements for the AI system.

Key Concepts Mentioned

AI system
AMIE (Video)
Gemini
low-latency dialogue
clinical reasoning
real-time audio-visual perception
primary care physicians (PCPs)
patient actors
Objective Structured Clinical Examination (OSCE)
clinical scenarios
history-taking
diagnosis
management
physical observation and examination
modality ablation
fine anatomical precision
subtle affective nuances
high-frequency movements

Claims Made by the Document

AMIE (Video) can perform at expert-level for real-time clinical video consultations.
Clinical evaluators rated AMIE (Video) on par or better than PCPs in history-taking, diagnosis, management, and physical observation and examination.
Patient actors preferred AMIE's approach to assessing and explaining conditions.

Evidence Used

Comparison made between AMIE (Video), its text-only counterpart AMIE (Text), and human PCPs in a study.
Clinical evaluators rated AMIE (Video) based on the results of the study.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Audio-visual interaction is the standard for patient-physician consultations, enabling natural communication and effective assessment of illness through non-verbal cues. While text-based AI has shown promise, it discards essential perceptual dimensions and limits patients who cannot articulate symptoms in writing. Early efforts to extend medical AI to audio-visual interaction have demonstrated feasibility but not reached clinician-level performance. Here, we provide the first demonstration of expert-level AI in real-time clinical video consultations using AMIE (Articulate Medical Intelligence Explorer) in a video configuration. AMIE (Video) is a Gemini-based multi-agent system integrating low-latency dialogue, clinical reasoning, and real-time audio-visual perception. To guide development, we established a taxonomy and automated evaluations for clinical audio-visual cues in telehealth settings. In a randomized Objective Structured Clinical Examination (OSCE) study with 30 primary care physicians (PCPs), 15 patient actors and 100 clinical scenarios, we compared AMIE (Video), its text-only counterpart AMIE (Text), and PCPs consulting via video. Clinical evaluators rated AMIE (Video) on par or better than PCPs in history-taking, diagnosis, management, and physical observation and examination. Patient actors preferred AMIE's approach to assessing and explaining conditions, while PCPs were preferred for rapport and partnership building. In modality ablation, patient actors preferred AMIE (Video)'s interface over text chat for communicative effectiveness, convenience, and feeling understood. Limitations remain in fine anatomical precision, subtle affective nuances, and high-frequency movements. While further research is needed before real-world translation, these results mark an important milestone toward AI systems capable of augmenting care across the sensory complexity of clinical practice.