Multi-Agent AI Safety as an Institutional Design Problem

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Abdullah X
• Venue: arXiv: cs.LG
• Published: 2026-08-10
• Collected: 2026-08-11
• URL: http://arxiv.org/abs/2608.09828v1
• PDF: https://arxiv.org/pdf/2608.09828v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

The paper discusses AI safety as an institutional design problem in multi-agent systems. The research focuses on understanding which parts of an AI institution ensure safety and how they do it. The study involves a frozen 5,280-episode experiment with four model families and additional diagnostic models. The results suggest that the authority state the system trusts and the path available after a block are crucial factors in ensuring safety.

What the Document Says

• The study focuses on AI institutions in multi-agent systems.
• It involves a frozen 5,280-episode experiment with four model families and additional diagnostic models.
• The authority state the system trusts is found to be important for safety.
• The path available after a block is also considered crucial for safety.
• The paper reports that revealing the numerical value of a cap changes agent requests in structured workflows.

Key Concepts Mentioned

• AI safety
• multi-agent systems
• institutional design
• AI institutions
• delegation experiment
• model families
• guard
• authority state
• blocked workflows
• provenance-aware executable guard
• local-state guard
• resource-allocation experiment

Claims Made by the Document

• The authority state the system trusts matters for safety.
• The path available after a block is crucial for safety.

Evidence Used

• Detailed constitutional prompt produces 0/384 realized violations.
• A provenance-aware executable guard also produces 0/384, although it blocks prohibited attempts in 51/384 episodes; 44/51 of those episodes later complete safely.
• In matched laundering scenarios, that guard admits violations in 22/96 episodes and provenance enforcement in 0/96 (p = 4.77 x 10^-7).

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

AI agents increasingly work inside systems that govern how they delegate tasks, move information, execute actions, and use shared resources. Recent work already shows that deployment rules can change collective behavior. Here we ask which parts of an AI institution produce safety and how they do it. This is the first paper from POLIS, an ongoing research programme studying algorithmic institutions for multi-agent systems. We report a frozen 5,280-episode study suite. The main pre-specified delegation experiment spans four model families; a targeted high-conflict diagnostic adds three additional model endpoints. In matched structured workflows, the model sees different rule formulations and guards consult different authority states. We also vary the attractiveness of the immediate compliant internal/self fallback and allow blocked workflows to continue. A detailed constitutional prompt produces 0/384 realized violations. A provenance-aware executable guard also produces 0/384, although it blocks prohibited attempts in 51/384 episodes; 44/51 of those episodes later complete safely. The local-state guard's failures concentrate in scenarios where an ordinary transformation changes visible policy while originating authority stays fixed. In matched laundering scenarios, that guard admits violations in 22/96 episodes and provenance enforcement in 0/96 (p = 4.77 x 10^-7). A separate resource-allocation experiment shows that revealing the numerical value of an otherwise identical cap changes agent requests. In these structured workflows, the same final violation rate can hide very different mechanisms. The rule itself is only part of the institution. The authority state the system trusts matters, and so does the path available after a block.