Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias

Source

arXiv

arxiv.org/abs/2607.11871v1

Read the source

Source

Plain-Language Brief

A research study presents an approach to understanding and mitigating bias in large language models (LLMs) used as judges, focusing on the hidden state of these models rather than just their input-output level. The study reports three findings: geometry, causal control, and operational implications. The geometry finding shows that biased inputs are displaced from a tight activation manifold compared to baseline inputs in a low-dimensional subspace specific to bias types. Causal control demonstrates that steering hidden states along this subspace can shift scoring both towards and away from biased results. Operational findings suggest that a simple linear projection using bias-direction features predicts judge failures on new benchmarks better than text-based alternatives.

What the Document Says

  • The research focuses on understanding and mitigating bias in LLMs used as judges, focusing on the hidden state of these models rather than just their input-output level.
  • Three findings are reported: geometry, causal control, and operational implications.
  • In the geometry finding, biased inputs are displaced from a tight activation manifold compared to baseline inputs in a low-dimensional subspace specific to bias types.
  • Causal control demonstrates that steering hidden states along this subspace can shift scoring both towards and away from biased results.
  • Operational findings suggest that a simple linear projection using bias-direction features predicts judge failures on new benchmarks better than text-based alternatives.

Key Concepts Mentioned

  • Large language models (LLMs)
  • Judging bias
  • Hidden state
  • Geometry
  • Causal control
  • Operational implications

Claims Made by the Document

  • Bias in LLMs used as judges can be understood at the representation level (hidden state).
  • Biased inputs are displaced from a tight activation manifold compared to baseline inputs in a low-dimensional subspace specific to bias types.
  • Steering hidden states along this subspace can shift scoring both towards and away from biased results.
  • A simple linear projection using bias-direction features predicts judge failures on new benchmarks better than text-based alternatives.

Evidence Used

  • Seven judges, seven bias types, and nine benchmarks are used in the study.
  • Three families of estimators are used to recover the low-dimensional subspace.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

The agent may not be able to fully understand the technical details of the study or its potential limitations. Further analysis by a human expert is recommended.

Open Questions

  • What are the potential applications of this approach in real-world scenarios?
  • How does this approach compare to existing methods for mitigating bias in LLMs used as judges?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The study proposes a different approach to understanding and mitigating bias in LLMs used as judges by focusing on the hidden state of these models. It reports three findings: geometry, causal control, and operational implications, providing evidence through the use of seven judges, seven bias types, and nine benchmarks.

Raw Abstract / Excerpt

Existing studies of LLM-as-judge scoring bias work predominantly at the input-output level: they perturb inputs, measure score deltas, and propose prompt-level mitigations. We argue that the same biases admit a representation-level account in the judge's hidden state, complementary to the input-output view and operationally useful in ways it does not afford. We report three findings, across seven judges, seven bias types, and nine benchmarks. Geometry: baseline judging inputs occupy a tight activation manifold while biased inputs are displaced along a low-dimensional, type-specific subspace that sharpens with depth and is recovered consistently by three families of estimators. Causal control: steering hidden states along this subspace drives scoring in both directions, forward shifts reproducing biased scoring on clean inputs and reverse shifts restoring baseline scoring on biased ones, while matched-norm random directions produce shifts an order of magnitude smaller. Operational: a simple linear projection onto the same bias-direction features anticipates judge failures on three entirely unseen benchmarks, substantially outperforming text-based alternatives. Reading bias as activation geometry, rather than as input-output noise, unifies geometric structure, causal control, and operational prediction within a single framework. The project page is available at https://xzx34.github.io/unfair-judge/