Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

An Explainable Agentic System for Detection of Conversational Scams with Summary-Based Memory

Source

arXiv

arxiv.org/abs/2607.11707v1

Read the source

Source

Plain-Language Brief

This research paper introduces an explainable AI system designed to detect conversational scams that can span multiple weeks or months and build trust to request money or sensitive information. The system, when tested on two datasets, was able to achieve high accuracy in identifying such scams. Two user studies suggest that users often feel uncertain when judging suspicious conversations, and the system is reported to increase trust, self-confidence, and perceived need for AI-based scam detection, with a usability score above the established benchmark.

What the Document Says

  • The paper presents an agentic system that extends single-message phishing detection and aims at detecting sophisticated conversational scams.
  • The system was tested on LoveFraud02 corpus and ConScamBench-278 datasets.
  • On the LoveFraud02 dataset, the conversation-level detector identified all conversational scams (83/83).
  • On ConScamBench-278, the system reached 97.8% accuracy (95% CI [95.4, 99.0]).
  • Two user studies were conducted with a total of 145 participants who reported often experiencing uncertainty when judging suspicious conversations.
  • In the user studies, users self-reported an increase in trust, self-confidence, and perceived need for AI-based scam detection (p < 0.001, Wilcoxon signed-rank).
  • The system received a System Usability Scale score of 74.7 (95% CI [72.5, 76.9]), exceeding the established usability benchmark.

Key Concepts Mentioned

  • Artificial Intelligence
  • Conversational scams
  • Phishing detection
  • User studies
  • System Usability Scale

Claims Made by the Document

  • The system is effective at identifying conversational scams.
  • Users often feel uncertain when judging suspicious conversations.
  • The system increases trust, self-confidence, and perceived need for AI-based scam detection.

Evidence Used

  • 100% phishing recall on isolated messages
  • identifying all conversational scams in LoveFraud02 corpus
  • 97.8% accuracy on ConScamBench-278
  • user study results

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Following the rapid progress of generative Artificial Intelligence, there is a growing threat posed by conversational scams. These scams often span over multiple weeks or months, gradually build trust and request for money or sensitive information. Existing scam-detection systems mainly focus on isolated messages, which renders them inadequate against this evolving threat. This paper extends single-message phishing detection and presents an explainable agentic system for detecting sophisticated conversational scams. It also introduces ConScamBench-278, an initial public multi-category benchmark for conversational scam detection spanning eight scam types, released to support reproducible evaluation and future expansion. On isolated messages the single-message detector attains 100% phishing recall, while the conversation-level detector identifies all conversational scams in the public LoveFraud02 corpus (83/83) and reaches 97.8% accuracy (95% CI [95.4, 99.0]) on ConScamBench-278. Two user studies (N = 100 and N = 45) further motivate the system: participants report frequently experiencing uncertainty when judging suspicious conversations. In an uncontrolled pre/post comparison, users self-reported trust, self-confidence, and perceived need for AI-based scam detection all increased (p < 0.001, Wilcoxon signed-rank). The system also receives a System Usability Scale score of 74.7 (95% CI [72.5, 76.9]), above the established usability benchmark.