One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Cesare Zavattari, Alessandro Tommasi, Giuseppe Prencipe
• Venue: arXiv: cs.AI
• Published: 2026-07-30
• Collected: 2026-07-31
• URL: http://arxiv.org/abs/2607.28317v1
• PDF: https://arxiv.org/pdf/2607.28317v1
• DOI: UNKNOWN

Plain-Language Brief

The paper discusses a scenario where a single human auditor has to evaluate multiple LLM (Language Learning Models) agents under a limited budget. The authors model this process as a noisy inspection over a two-level Gaussian copula. They find that there is a miscalibration threshold, $δ^*$, above which relying on confidence-ranked auditing becomes worse than random selection. The study reveals some unexpected results: the miscalibration threshold increases as the budget decreases, and cross-family correlation is not necessarily low - shared difficulty among models dominates lineage. They also introduce a criterion for 'vacuous' oversight and confirm their findings by testing on recorded traces.

What the Document Says

• A human auditor evaluates N LLM agents under budget B
• Confidence may be adversarially miscalibrated and correlated
• The authors model this as a budgeted noisy inspection over a two-level Gaussian copula
• They locate the miscalibration threshold $δ^*$ where confidence-ranked auditing is worse than random
• As budget shrinks, $δ^*$ rises
• Cross-family correlation is not low - shared difficulty dominates lineage
• Five open-weight LLMs show operationally useless (near-constant) confidence
• Their confidence point estimates are at or beyond the flip, but CIs straddle it
• A proprietary model is informative and lands below it
• They introduce a quantitative criterion for 'vacuous' oversight
• Testing on recorded traces confirms their ordering

Key Concepts Mentioned

• human auditor
• LLM agents
• budgeted noisy inspection
• miscalibration threshold
• Gaussian copula
• confidence-ranked auditing

Claims Made by the Document

• Confidence-ranked auditing is worse than random when the miscalibration threshold $δ^*$ is exceeded
• The miscalibration threshold $δ^*$ increases as budget shrinks
• Cross-family correlation is not low - shared difficulty dominates lineage

Evidence Used

• Modeling the process as a budgeted noisy inspection over a two-level Gaussian copula
• Finding that confidence-ranked auditing is worse than random when the miscalibration threshold $δ^*$ is exceeded in simulations

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

A single human must audit $N$ LLM agents under a budget of $B \ll N$ audits per round, guided by self-reported confidence that may be adversarially miscalibrated and by correlated errors. We model this as budgeted noisy inspection over a two-level Gaussian copula and locate the miscalibration threshold $δ^$ past which confidence-ranked auditing is \emph{worse} than random. Two a-priori expectations reverse: $δ^$ \emph{rises} as the budget shrinks, and cross-family correlation is not low---shared difficulty dominates lineage. Five open-weight LLMs show operationally useless (near-constant) confidence, point estimates at or beyond the flip though CIs straddle it; a proprietary model is informative and lands below it. We give a quantitative criterion for \emph{vacuous} oversight, and replaying policies on recorded traces confirms the ordering.