MuPPET: A Benchmark for Contextual Privacy of LLM Assistants in Multi-Party Conversations

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Elena Sofia Ruzzetti, Cornelius Emde, Sangdoo Yun, Seong Joon Oh, Martin Gubri
• Venue: arXiv: cs.CL
• Published: 2026-06-22
• Collected: 2026-06-23
• URL: http://arxiv.org/abs/2606.23217v1
• PDF: https://arxiv.org/pdf/2606.23217v1
• DOI: UNKNOWN

Plain-Language Brief

The researchers introduce a new benchmark called MuPPET to measure contextual privacy in multi-party conversations involving LLM agents handling sensitive personal data for multiple users. They found that models leak more private information in multi-party settings than in one-to-one evaluations, with frontier models and smaller open-weights models being particularly vulnerable.

What the Document Says

• LLM agents are used in multi-party environments for handling sensitive personal data.
• When an LLM agent discloses private information in a multi-party conversation, it reaches every group member simultaneously.
• Existing contextual privacy benchmarks only consider single-interlocutor settings, leaving multi-party privacy risks unmeasured.
• The researchers introduce MuPPET (Multi-Party Privacy Exposure Testing), a benchmark for contextual privacy in multi-party conversations.
• Experiments show that models leak substantially more private information in multi-party settings than one-to-one evaluations suggest.
• Frontier models and smaller open-weights models are vulnerable to leaks in multi-party settings.
• Existing contextual privacy defenses offer only partial protection, degrade utility, and do not resolve the underlying party-tracking problem.

Key Concepts Mentioned

• LLM agents
• multi-party conversations
• sensitive personal data
• contextual privacy
• benchmark
• MuPPET
• frontier models
• open-weights models
• leaks

Claims Made by the Document

• Multi-party privacy risks are not measured by existing contextual privacy benchmarks.
• Models leak more private information in multi-party settings than one-to-one evaluations suggest.
• Frontier models and smaller open-weights models are vulnerable to leaks in multi-party settings.
• Existing contextual privacy defenses offer only partial protection, degrade utility, and do not resolve the underlying party-tracking problem.

Evidence Used

• The researchers present MuPPET as a solution to address the lack of measurement for multi-party privacy risks.
• The experiments conducted by the researchers demonstrate the increased leaks in multi-party settings compared to one-to-one evaluations.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

• The authors do not provide specific details about the methods used for their experiments.

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The agent is only summarizing the provided text and does not have additional information about the research or its methodology.
• The agent's interpretation may be influenced by its programming and does not reflect the authors' intended meaning in all cases.

Open Questions

• What type of LLM agents are being referred to?
• What specific methods were used for the experiments conducted by the researchers?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The text does not specify the type of LLM (Language Learning Model) agents being referred to.
• The research paper provides no information on the specific methods used for the experiments.

Raw Abstract / Excerpt

LLM agents are increasingly deployed in multi-party environments, handling sensitive personal data on behalf of individual users, for instance in group chats. When such an agent discloses private information, it reaches every group member at once. This risk is structurally harder to control than in one-to-one settings, as every piece of private information must be appropriate for every recipient in the group. Yet all existing contextual privacy benchmarks consider only single-interlocutor settings, leaving multi-party privacy risks unmeasured. We introduce MuPPET (Multi-Party Privacy Exposure Testing), a benchmark for contextual privacy in multi-party conversations. Our experiments show that models leak substantially more in multi-party settings than one-to-one evaluations suggest. Frontier models are vulnerable, and smaller open-weights models, often preferred for local deployment with sensitive data, even more so. Existing contextual privacy defences offer only partial protection, degrade utility, and do not resolve the underlying party-tracking problem.