Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

An Explainable Agentic System for Detection of Conversational Scams with Summary-Based Memory

Source

arXiv

arxiv.org/abs/2607.11707v1

Read the source

Source

Plain-Language Brief

The paper presents an 'explainable agentic system' for detecting conversational scams over multiple messages, which existing systems often fail to do effectively. The new system achieves high accuracy in both single-message and conversation-level detection. Two user studies suggest that users feel more confident and trust the system after using it, with a usability score above the established benchmark.

What the Document Says

  • The paper introduces an explainable agentic system for detecting conversational scams.
  • The system achieves 100% phishing recall on isolated messages and identifies all conversational scams in the LoveFraud02 corpus.
  • On ConScamBench-278, the system reaches 97.8% accuracy with a confidence interval of [95.4, 99.0].
  • Two user studies were conducted to motivate the system, with participants reporting increased trust, self-confidence, and perceived need for AI-based scam detection.
  • The system received a System Usability Scale score of 74.7 with a confidence interval of [72.5, 76.9].

Key Concepts Mentioned

  • Conversational scams
  • Artificial Intelligence
  • Explainable agentic system
  • Single-message phishing detection
  • ConScamBench-278
  • LoveFraud02 corpus

Claims Made by the Document

  • The new system can detect conversational scams effectively.
  • Users feel more confident and trust the system after using it.
  • The system has a high usability score.

Evidence Used

  • The accuracy rates provided for the system's performance.
  • The results of the two user studies.

Methods / Approach

  • Developing an explainable agentic system
  • Conducting user studies

Limitations Stated by the Authors

The paper does not provide details on how the user studies were conducted or what specific aspects of the system users found most useful.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The agent's understanding is limited to the information provided in the text. Further research may be needed to understand the system's internal workings and potential weaknesses.

Open Questions

  • How does the new system differ from existing scam-detection systems?
  • What specific aspects of the system do users find most useful or confusing?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper does not provide information on how the system's performance compares to other state-of-the-art methods.
  • The user studies might have some biases due to self-reported data and small sample sizes.

Raw Abstract / Excerpt

Following the rapid progress of generative Artificial Intelligence, there is a growing threat posed by conversational scams. These scams often span over multiple weeks or months, gradually build trust and request for money or sensitive information. Existing scam-detection systems mainly focus on isolated messages, which renders them inadequate against this evolving threat. This paper extends single-message phishing detection and presents an explainable agentic system for detecting sophisticated conversational scams. It also introduces ConScamBench-278, an initial public multi-category benchmark for conversational scam detection spanning eight scam types, released to support reproducible evaluation and future expansion. On isolated messages the single-message detector attains 100% phishing recall, while the conversation-level detector identifies all conversational scams in the public LoveFraud02 corpus (83/83) and reaches 97.8% accuracy (95% CI [95.4, 99.0]) on ConScamBench-278. Two user studies (N = 100 and N = 45) further motivate the system: participants report frequently experiencing uncertainty when judging suspicious conversations. In an uncontrolled pre/post comparison, users self-reported trust, self-confidence, and perceived need for AI-based scam detection all increased (p < 0.001, Wilcoxon signed-rank). The system also receives a System Usability Scale score of 74.7 (95% CI [72.5, 76.9]), above the established usability benchmark.