Shortcut Sensitivity in Agent-Safety Evaluations

Source facts

The paper audits label predictability, observational guard association, and paired decision sensitivity in LinuxArena trajectories and TS-Bench-derived datasets. The abstract reports performance changes under domain holdout and interaction reweighting, and reports that rewriting an information marker or using opaque tool aliases changed blocking rates in a frozen AgentDojo-Traj screen while unsafe calls remained blocked.

Analysis

The work treats benchmark representation as an intervention on the guard, not merely as a passive measurement choice. Domain holdouts, evaluation-unit metrics, and placebo-controlled paired interventions may therefore be important when safety-guard accuracy is used as evidence of generalization.

Uncertainty/Doubt

• This is an arXiv preprint. The reported shortcut effects, exploratory ranges, and model-specific replication boundaries were not independently reproduced here.