DUMA-Bench: A Dual-Control Multi-Agent Benchmark for Evaluating LLM Agent Security

Source facts

DUMA-Bench evaluates agent security when both the agent and user can influence shared environment state. It extends tau-squared bench with adversarial environments covering eight vulnerability classes, including RAG poisoning, cross-agent manipulation, and unsafe output handling. The preprint evaluates 14 models from five model families across eight domains and user-behavior regimes.

Analysis

The benchmark makes the user-agent-environment interaction part of the security test surface. That is a useful complement to static or passive-user evaluations because realistic deployments include adversarial, accidental, and changing user behavior.

Uncertainty/Doubt

• This is an arXiv preprint with a stated ACL ARR reference; the benchmark design and reported attack-success changes were not independently reproduced here.