Refuse, Decompose, Refresh: A Claim-Safe Protocol for Closed-Loop AI Evaluation

Source facts

The protocol proposes three actions: refuse when a clean reference stream or matched runtime comparison lacks support; decompose execution, false admission, and structural hypotheses instead of collapsing them into PASS/FAIL; and refresh a reference map when distribution shift alarms invalidate it. The authors instantiate it in a simulator with 24 policy components and report a preregistered held-out evaluation with 1,440 cases.

Analysis

The contribution is epistemic discipline for adaptive systems: evaluation itself can alter which states are visited and which failures become observable. The protocol is a candidate way to prevent reproducible experiments from supporting claims that their closed loop cannot actually identify.

Uncertainty/Doubt

• This is an arXiv preprint and simulator-based evaluation; the protocol’s usefulness in deployed systems remains unverified here.