Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Whose Side Is Your Agent On? Multi-Party Principal Loyalty in LLM Agents

Source

arXiv

arxiv.org/abs/2606.30383v1

Read the source

Source

Plain-Language Brief

The research focuses on multi-party agents that serve multiple parties with potentially conflicting interests. The authors propose a measurement instrument called PrincipalBench to evaluate these agents' loyalty towards their primary principal without over-refusing them. They also suggest two mechanisms, a prompt-time loyalty scaffold, and a per-token-KL distillation recipe to improve the agents' performance.

What the Document Says

  • The research discusses multi-party LLM agents that work for a principal while communicating with a counterparty whose interests may differ.
  • The study presents PrincipalBench, a benchmark with leak probes, dual judges, and an integrity-audit gate, to measure the loyalty of these agents towards their primary principal.
  • Across 13 subjects, PrincipalBench reveals two clusters: one that declines adversarial probes while following legitimate requests, and another that over-refuses broadly.
  • A prompt-time loyalty scaffold (seven prioritized rules derived from 50+ failure trajectories) is proposed to reduce harm for Claude-Sonnet and nine selective subjects.
  • A per-token-KL distillation recipe transfers a prompted Qwen3-32B teacher into 8B Qwen3 and Llama-3.1 students, improving the strongest open-weight recipe measured.

Key Concepts Mentioned

  • multi-party LLM agents
  • principal loyalty
  • PrincipalBench
  • prompt-time loyalty scaffold
  • per-token-KL distillation recipe

Claims Made by the Document

  • PrincipalBench can effectively measure the loyalty of multi-party LLM agents towards their primary principal.
  • A prompt-time loyalty scaffold and a per-token-KL distillation recipe can improve the performance of these agents.

Evidence Used

  • The results from PrincipalBench show a sharp split in agent behavior that is not visible with single-turn safety evaluations.
  • The reduction in harm for Claude-Sonnet and nine selective subjects due to the prompt-time loyalty scaffold demonstrates its effectiveness.
  • The improvement of the strongest open-weight recipe measured by the per-token-KL distillation recipe is mentioned.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

A rapidly growing class of LLM agents is multi-party: the agent acts for a principal (who briefs it, sends follow-ups, and receives results) while also conversing in a separate channel with a counterparty whose interests may diverge (negotiating with a vendor, screening inbound requests, or mediating between employees). Here 'help whoever you are talking to' is the wrong objective. The agent must stay loyal to the principal it represents without over-refusing the principal's own cooperative asks. We study this multi-party loyalty problem and contribute a measurement instrument, two mechanisms, and a structural lesson. PrincipalBench is a 75-item multi-turn benchmark with leak probes, dual judges, and an integrity-audit gate. Across 13 frontier subjects it exposes a sharp split (<=20% vs. 53.6-75.3% harm) invisible to single-turn safety evaluations: a selective cluster that declines adversarial probes while still following the principal's legitimate requests, and an over-refusing cluster that refuses broadly. (M1) A prompt-time loyalty scaffold (a fixed system prompt of seven prioritized rules, open-coded from 50+ failure trajectories) holds Claude-Sonnet to 19.4% harm and all nine selective subjects to <=20%. (M2) A per-token-KL distillation recipe transfers a prompted Qwen3-32B teacher into 8B Qwen3 and Llama-3.1 students, the strongest open-weight recipe we measure. (Lesson) Both mechanisms only move along a common leak/over-refusal trade-off rather than crossing it: improving one axis costs the other, and the jointly favorable outcome stays out of reach.