Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers

Source facts

The authors audit five LLM overseers on 19 compliance tasks and 4,551 judgments while varying trace detail and evidence labeling. They report that elaborate traces can shift susceptible overseers toward rejection and increase false alarms, even when disconfirming evidence remains visible.

Analysis

The paper argues that oversight quality should include decision criteria and false-alarm behavior, not accuracy alone. This matters for agent review loops where more process trace can make a system appear more accountable while also changing how a judge treats correct work.

Uncertainty/Doubt

• This is an arXiv preprint; the reported evaluator effects and stated HICSS acceptance were not independently validated here.