A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Praveen Reddy, Charuta Mandke, Suvrankar Datta, Sarah Khan, Siddharth Reddy Anthireddy, Shitij Arora, Vishal Singh
• Venue: arXiv: cs.CL
• Published: 2026-08-12
• Collected: 2026-08-13
• URL: http://arxiv.org/abs/2608.12138v1
• PDF: https://arxiv.org/pdf/2608.12138v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

A research paper evaluates VITA, a clinical knowledge retrieval system built for low- and middle-income settings, on a medical benchmark. In comparison to various frontier large language models (LLMs), including GPT-4.1, o4-mini, Gemini 3.1 Pro, Claude Sonnet 4.6, GPT-5.4, GPT-5.5, Claude Opus 4.8, and Grok 4.3, VITA performed better or comparably on accuracy and completeness. However, its communication scores were lower.

What the Document Says

• The study compares a clinical RAG system called VITA with several frontier LLMs on an open benchmark called HealthBench.
• VITA retrieves information from a curated corpus specific to India and low- and middle-income settings.
• On 4,023 English-language HealthBench questions, VITA outperformed or matched other LLMs in terms of accuracy and completeness, except for GPT-5.5 with which it was statistically indistinguishable.
• VITA's communication scores were lower compared to other models.
• The study also tested the robustness of VITA against newer models by using a 500-question subset and a neutral judge.

Key Concepts Mentioned

• clinical RAG system
• VITA
• large language models (LLMs)
• HealthBench
• accuracy
• completeness
• communication scores

Claims Made by the Document

• VITA outperformed or matched other LLMs on an open benchmark.
• VITA's advantages in accuracy and completeness persisted under the neutral judge.

Evidence Used

• On 4,023 HealthBench questions, VITA ranked first with 51.9% of possible rubric points.
• VITA led on points-weighted score and won the most questions in the comparison with newer models.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

General-purpose large language models (LLMs) have recently been reported to match or exceed specialized clinical AI tools on medical benchmarks, but such comparisons draw on a narrow set of systems and on benchmarks developed largely in high-income settings. We evaluate VITA, a retrieval-augmented generation (RAG) system purpose-built for contextual knowledge retrieval in India and other low- and middle-income (LMIC) settings. VITA retrieves from a curated corpus of disease-specific guidelines, India-specific antimicrobial resistance data, national formulary constraints, and resource-limited care protocols; its architecture and corpus are proprietary, but the benchmark, the physician-written rubrics, and our full response and scoring outputs are public for independent verification. On 4,023 English-language HealthBench questions (80.5% of the benchmark), scored with a GPT-4.1 judge, VITA ranked first with 51.9% of possible rubric points, ahead of GPT-5.4 (46.1%), o4-mini (44.3%), Gemini 3.1 Pro (42.6%), and Claude Sonnet 4.6 (37.3%), and scored highest on 45.4% of questions. To test robustness to newer models and judge lineage, a 500-question subset was re-run against current-generation models (GPT-5.5, Claude Opus 4.8, Gemini 3.5 Pro, Grok 4.3) and graded by a neutral open-weight judge (DeepSeek-V4-Pro) sharing no lineage with any system tested. Here the gap narrowed to parity: VITA and GPT-5.5 were statistically indistinguishable on mean per-question score, while VITA led on points-weighted score and won the most questions. VITA's advantages in accuracy and completeness persisted under the neutral judge; its communication scores were lower. These results indicate that a purpose-built clinical RAG system remains competitive with frontier LLMs on an open benchmark, consistent with corpus specificity as a design variable that improves grounding at some cost to communication polish.