When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Zongyou Yang, Yinghan Hou, Xiaokun Yang
• Venue: arXiv: cs.CL
• Published: 2026-07-09
• Collected: 2026-07-10
• URL: http://arxiv.org/abs/2607.08535v1
• PDF: https://arxiv.org/pdf/2607.08535v1
• DOI: UNKNOWN

Plain-Language Brief

The paper investigates how changes in the evaluators used by large language models (LLMs) to score candidate responses can impact the model's performance and reliability. The study examines two common upgrade paths for LLMs: scaling Qwen3 dense judges from 1.7B to 32B parameters and transitioning across MiniMax M2-M2.7 APIs. The research finds that judge upgrades are not always interchangeable, with Qwen3 1.7B to 4B providing a consistent improvement, while MiniMax updates do not. The study also highlights the persistence of bias in stronger judges and suggests that repeating samples from juries add little when errors are correlated. Structured debate can significantly impact decisions, but without parsers and fallback logs, it's unclear if shifts are due to deliberation.

What the Document Says

• There is a problem of evaluator-replacement ambiguity in LLM-as-judge scores where the score can change even with fixed candidate responses due to changes in the evaluator.
• Two common upgrade paths for LLMs are examined: scaling Qwen3 dense judges from 1.7B to 32B parameters and moving across MiniMax M2-M2.7 APIs.
• Qwen3 1.7B to 4B provides a robust adjacent gain, while MiniMax adjacent releases do not.
• Stronger judges reduce but do not remove position and verbosity bias.
• Repeated-sample juries add little when errors are correlated.
• Structured debate can move decisions substantially, but without parser and fallback logs, shifts cannot be attributed to deliberation.
• The paper suggests that LLM-as-judge reports should include dataset slices, bias probes, error-dependence estimates, and protocol audit trails.

Key Concepts Mentioned

• LLM-as-judge
• evaluator-replacement ambiguity
• Qwen3
• MiniMax
• bias
• structured debate

Claims Made by the Document

• Evaluator changes can impact LLM-as-judge scores.
• Qwen3 1.7B to 4B provides a robust improvement compared to other judge upgrades.
• MiniMax adjacent releases do not provide consistent improvements compared to Qwen3 1.7B to 4B.
• Stronger judges reduce but do not remove position and verbosity bias.
• Repeated-sample juries add little when errors are correlated.
• Structured debate can significantly impact decisions.
• LLM-as-judge reports should include certain details for improved transparency.

Evidence Used

• The study compares the impacts of two upgrade paths on LLM performance.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed. We treat this evaluator-replacement ambiguity as a measurement-validity problem. Across four judgment datasets, we compare two upgrade paths available in practice: scaling Qwen3 dense judges from 1.7B to 32B parameters and moving across MiniMax M2-M2.7 released APIs. The main pattern is that judge upgrades are not interchangeable: only Qwen3 1.7B to 4B gives a robust adjacent gain, while MiniMax adjacent releases do not. Stronger judges reduce but do not remove position and verbosity bias. Repeated-sample juries add little when errors are correlated. Structured debate can move decisions substantially, but without parser and fallback logs those shifts cannot be attributed to deliberation. We argue that LLM-as-judge reports should include dataset slices, bias probes, error-dependence estimates, and protocol audit trails.