Verdict Without the Rule: Diagnosing and Auditing Regulatory Rule Sensitivity in LLM Compliance Systems

Source facts

The preprint tests regulatory-rule sensitivity by deleting, swapping, or negating a rule while holding the case fixed across five models and 20 domains. It reports that verdicts are often invariant to large rule perturbations. The abstract reports a guard model at 51% rule sensitivity compared with 90–92% for the general-purpose models, and argues that accuracy alone does not establish grounding in the supplied rule.

Analysis

Counterfactual rule perturbations provide a direct audit for whether a compliance system uses the governing text it cites. The reported gap between the guard model and general-purpose models suggests that rule sensitivity should be measured separately from answer accuracy in regulated deployments.

Uncertainty/Doubt

• This is an arXiv preprint. The model set, perturbation design, and reported percentages were not independently reproduced here.