Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

LLM-Based Test Oracles: Source-of-Authority Taxonomy -- A Systematic Literature Review

Source

arXiv

arxiv.org/abs/2607.05031v1

Read the source

Source

Plain-Language Brief

A study on Large Language Models (LLMs) used to determine correct behavior in tests, identifies a gap in understanding the source of these models' authority. The study conducts a systematic literature review and finds that about half of the studies use specification-derived authority, while the remaining 26 have no specified source at all. The research highlights the need for further investigation on how these oracles are evaluated and their potential failures.

What the Document Says

  • The study focuses on Large Language Models (LLMs) used as test oracles.
  • A clear account of where these oracles draw their authority is missing.
  • Prior studies organize the area by oracle form or LLM technique, but not by the source of the verdict's authority.
  • The study presents a systematic literature review under PRISMA 2020 guidelines.
  • From 2436 records, 54 studies were included after an LLM pre-filter, independent dual human screening, and full-text assessment.
  • The analysis of the 54 studies is done along three axes: source of oracle's authority, its form, and the mechanism that adjudicates it.
  • Specification-derived authority is the most common single source, covering about half of the studies.
  • The remaining 26 studies reach a verdict with no specification at all.
  • The study further reports how these oracles are evaluated and how they fail.
  • The research agenda includes investigating empty regions of the taxonomy.

Key Concepts Mentioned

  • Large Language Models (LLMs)
  • Test Oracles
  • Source-of-Authority Taxonomy
  • Systematic Literature Review
  • PRISMA Guidelines

Claims Made by the Document

  • A clear account of where LLM test oracles draw their authority is missing.
  • Half of the studies use specification-derived authority.

Evidence Used

  • The study states that a clear account of the source of authority for LLM test oracles is missing.
  • The study reports that about half of the studies use specification-derived authority.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large language models (LLMs) are increasingly used to produce test oracles, the part of a test that decides whether observed behavior is correct. Yet a clear account of where these oracles draw their authority is missing. Prior secondary studies organize the area by oracle form or by LLM technique. None organizes it by the source of the verdict's authority, the property that governs how far a verdict can be trusted. This article presents a systematic literature review, conducted and reported under the PRISMA 2020 guidelines. From 2,436 records, an LLM pre-filter followed by independent dual human screening (reviewer agreement, a Cohen's kappa of 0.79) and full-text assessment yielded 54 included studies. We analyze these along three axes: the source of an oracle's authority, the form it takes, and the mechanism that adjudicates it. We characterize the landscape of domains, languages, models, and adaptation strategies. Specification-derived authority, though the most common single source, covers about half of the studies (28 of 54). The remaining 26 reach a verdict with no specification at all. The source of authority and the adjudication mechanism cross-cut: the same source is checked by several mechanisms and one mechanism serves several sources, so a label such as LLM-as-a-judge names a mechanism rather than a basis for trust. We further report how these oracles are evaluated and how they fail, and read the sparse and empty regions of the taxonomy as a research agenda. The protocol, search query, and per-study coding sheet are released as supplementary material.