Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Poster: Exploring the Limits of Audio-Based Detection of Turkish Phone Call Scams

Source

arXiv

arxiv.org/abs/2606.24523v1

Read the source

Source

Plain-Language Brief

This research explores the use of large language models (LLMs) in detecting scam phone calls in Turkish, a low-resource language where such detection is particularly challenging due to limited annotated data and technological defenses. The study introduces a new public multi-modal dataset of 100 Turkish audio-transcript pairs for both scam and benign conversations. Seven LLMs were tested under three input conditions: raw audio, automatic speech-to-text transcripts, and human-corrected or uncorrected transcripts. The findings indicate that transcript-based inputs outperform direct audio processing, and there is no significant difference in performance between human-corrected and uncorrected transcripts.

What the Document Says

  • The research investigates the use of large language models for detecting scam phone calls in Turkish, a low-resource language.
  • A new public multi-modal dataset of 100 Turkish audio-transcript pairs is introduced.
  • Seven LLMs were tested under three input conditions: raw audio, automatic speech-to-text transcripts, and human-corrected or uncorrected transcripts.
  • Transcript-based inputs outperform direct audio processing.
  • There is no significant difference in performance between human-corrected and uncorrected transcripts.

Key Concepts Mentioned

  • scam phone calls
  • Turkish language
  • large language models (LLMs)
  • multi-modal dataset
  • transcripts

Claims Made by the Document

  • LLMs can support scam detection in Turkish.
  • Transcript-based inputs outperform direct audio processing.

Evidence Used

  • Introduction of the new multi-modal dataset and testing results for seven LLMs under three input conditions.

Methods / Approach

  • Testing seven large language models on a newly introduced public multi-modal dataset

Limitations Stated by the Authors

  • The study does not provide information about the specific metrics used to evaluate the performance of the LLMs.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The findings are based on one specific dataset and may not generalize to other datasets or languages.
  • The research focuses on phone scams, and it's unclear how these findings apply to other types of scams.

Open Questions

  • What specific metrics were used to evaluate the performance of the LLMs?
  • How do these findings apply to other types of scams or languages?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The study does not specify the exact evaluation metrics employed.
  • The research is limited to phone scams in Turkish.

Raw Abstract / Excerpt

Scam phone calls exploit vulnerable communities worldwide, yet research on detection has focused almost exclusively on English and other high-resource languages. In low-resource settings such as Turkish, detection is especially difficult, as annotated data is scarce and technological defenses remain limited. This research investigates how large language models (LLMs) can support scam detection in Turkish by introducing the first public multi-modal dataset of 100 aligned audio-transcript pairs of scam and benign conversations. We evaluate seven LLMs spanning three model families: Gemini 2.5 (Flash, Flash-Lite, Pro), GPT-4o, and Qwen (Max, Plus, Turbo), under three input conditions: raw audio, automatic speech-to-text transcripts, and transcripts refined by a native speaker. Our results suggest that transcript-based inputs consistently outperform direct audio processing, while human-corrected and uncorrected transcripts perform comparably. By centering a low-resource language and real world threat, this work highlights the urgent need for culturally and linguistically inclusive AI safety research and more robust multi-modal systems for fraud prevention.