PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety

Source facts

PASTABench evaluates safety monitoring over multi-turn agent trajectories, covering 1,139 trajectories, five risk categories, and 13 subcategories. It proposes an Optimal Intervention Window based on the earliest useful signal and the eventual trigger turn. The preprint evaluates 16 LLMs and reports that the best model reached 40.74% on optimal timing; it also reports that some smaller-model scores collapse when hazard vocabulary is neutralized.

Analysis

The benchmark shifts safety evaluation from detecting a completed unsafe action to deciding when intervention should occur during a trajectory. This creates a practical systems target for monitors: early enough to prevent harm, but not so early that normal work is unnecessarily interrupted.

Uncertainty/Doubt

• This is an arXiv preprint and identifies an EMNLP 2026 comment in its metadata. The benchmark construction, optimal-window labels, model comparisons, and conference status were not independently verified here.