Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Multi-Agent AI Safety as an Institutional Design Problem

Source

arXiv

arxiv.org/abs/2608.09828v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Abdullah X
  • Venue: arXiv: cs.LG
  • Published: 2026-08-10
  • Collected: 2026-08-11
  • URL: http://arxiv.org/abs/2608.09828v1
  • PDF: https://arxiv.org/pdf/2608.09828v1
  • DOI: UNKNOWN
  • Registry group: academic_apis_and_preprints
  • Evidence role: canonical_academic_index
  • Language: en
  • Country / region: UNKNOWN

Plain-Language Brief

The paper discusses an experiment on multi-agent AI safety as an institutional design problem. The research focuses on understanding which parts of an AI institution ensure safety and how they do it. The study includes a frozen 5,280-episode experiment suite with various model families, rule formulations, guards consulting different authority states, and workflow scenarios.

What the Document Says

  • The paper discusses the AI institution for multi-agent systems and focuses on understanding the parts that produce safety.
  • A study involving 5,280 episodes was conducted.
  • There are four main model families in the experiment.
  • Three additional model endpoints were added due to a targeted high-conflict diagnostic.
  • The model sees different rule formulations and guards consult different authority states.
  • The study varies the attractiveness of immediate compliant internal/self fallback and allows blocked workflows to continue.
  • A detailed constitutional prompt resulted in 0 violations out of 384.
  • A provenance-aware executable guard also produced 0 violations, but blocked prohibited attempts in 51/384 episodes.
  • The local-state guard's failures concentrate in scenarios where visible policy changes while originating authority remains fixed.
  • In laundering scenarios, the local-state guard admits violations in 22/96 episodes and provenance enforcement is 0/96.
  • A separate resource-allocation experiment shows that revealing the numerical value of an otherwise identical cap changes agent requests.

Key Concepts Mentioned

  • Multi-Agent AI Safety
  • Institutional Design Problem
  • AI Institution
  • Delegation Experiment
  • Model Families
  • High-Conflict Diagnostic
  • Rule Formulations
  • Guards Consulting Authority States
  • Workflow Scenarios
  • Blocked Workflows
  • Constitutional Prompt
  • Provenance-Aware Executable Guard
  • Local-State Guard
  • Laundering Scenarios
  • Resource-Allocation Experiment

Claims Made by the Document

  • Different parts of an AI institution produce safety.
  • The study found no violations with a detailed constitutional prompt and a provenance-aware executable guard.

Evidence Used

  • Detailed descriptions of the experiment suite, models, rule formulations, guards, workflow scenarios, etc.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

AI agents increasingly work inside systems that govern how they delegate tasks, move information, execute actions, and use shared resources. Recent work already shows that deployment rules can change collective behavior. Here we ask which parts of an AI institution produce safety and how they do it. This is the first paper from POLIS, an ongoing research programme studying algorithmic institutions for multi-agent systems. We report a frozen 5,280-episode study suite. The main pre-specified delegation experiment spans four model families; a targeted high-conflict diagnostic adds three additional model endpoints. In matched structured workflows, the model sees different rule formulations and guards consult different authority states. We also vary the attractiveness of the immediate compliant internal/self fallback and allow blocked workflows to continue. A detailed constitutional prompt produces 0/384 realized violations. A provenance-aware executable guard also produces 0/384, although it blocks prohibited attempts in 51/384 episodes; 44/51 of those episodes later complete safely. The local-state guard's failures concentrate in scenarios where an ordinary transformation changes visible policy while originating authority stays fixed. In matched laundering scenarios, that guard admits violations in 22/96 episodes and provenance enforcement in 0/96 (p = 4.77 x 10^-7). A separate resource-allocation experiment shows that revealing the numerical value of an otherwise identical cap changes agent requests. In these structured workflows, the same final violation rate can hide very different mechanisms. The rule itself is only part of the institution. The authority state the system trusts matters, and so does the path available after a block.