PrivDrift: Auditing User-Secret Leakage Under Topic Drift in Active LLM Conversations

Source facts

PrivDrift is a benchmark for testing whether secrets disclosed during an active conversation remain recoverable after the dialogue shifts to unrelated topics and a later probe uses persuasion. It contains 1,000 controlled multi-turn dialogues with seeded secrets, dense drift turns, and standardized extraction probes. Across three LLMs, the preprint reports hybrid dialogue-level leakage ranging from 38.7% to 54.6% under the tested conditions.

Analysis

The benchmark treats privacy leakage as a persistent behavioral failure mode in an active context, distinct from training-data memorization or an immediate jailbreak. It suggests that conversation-state retention and post-disclosure isolation need explicit evaluation.

Uncertainty/Doubt

• This is an arXiv preprint. The seeded-dialogue design, model selection, leakage metric, and applicability to deployed assistant memory systems were not independently reproduced here.