Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection

Source

arXiv

arxiv.org/abs/2607.06254v1

Read the source

Source

Plain-Language Brief

A study introduces VendorBench-100, a cross-paradigm benchmark for deepfake image detection that evaluates different models using a common protocol. The benchmark includes a diverse set of edge cases and uses the Matthews correlation coefficient (MCC) and ROC-AUC to rank models.

What the Document Says

  • The text discusses a new benchmark called VendorBench-100 for deepfake image detection.
  • VendorBench-100 evaluates 36 representative models using a single adversarial 100-image corpus, a unified output schema, and a common evaluation framework.
  • The benchmark includes eight edge-case families such as face swaps, text-to-video stills, AI photo edits, avatar compositing, opaque-provenance images, and compressed research frames.
  • Ranking is primarily done using the Matthews correlation coefficient (MCC), with ROC-AUC reported as a secondary measure.
  • The text mentions that commercial APIs achieve the strongest median performance, followed by vision LLMs and open-source detectors.
  • Individual open-source models remain competitive with the best vision LLMs.
  • There is a consistent divergence between ranking ability (ROC-AUC) and operating-point quality (MCC), demonstrating that strong score discrimination does not necessarily produce reliable default-threshold decisions.
  • The benchmark's central finding is this metric disagreement, rather than any single leaderboard ranking.

Key Concepts Mentioned

  • deepfake image detection
  • VendorBench-100
  • commercial APIs
  • zero-shot vision-language models (LLMs)
  • open-source detectors
  • Matthews correlation coefficient (MCC)
  • ROC-AUC

Claims Made by the Document

  • Commercial APIs achieve the strongest median performance for deepfake image detection.
  • Individual open-source models remain competitive with the best vision LLMs in deepfake image detection.

Evidence Used

  • The text states that commercial APIs have the strongest median performance and that individual open-source models are competitive with the best vision LLMs.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Deepfake image detection is currently served by three fundamentally different paradigms: commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors. Despite their widespread use, these paradigms are rarely evaluated under a common protocol, making direct comparison difficult. We introduce VendorBench-100, a cross-paradigm benchmark that evaluates 36 representative models using a single adversarial 100-image corpus, a unified output schema, and a common evaluation framework. To ensure reliable assessment under the corpus's intentional class imbalance, models are ranked primarily by the Matthews correlation coefficient (MCC), with ROC-AUC reported as a threshold-independent measure of ranking ability. Rather than maximizing dataset size, VendorBench-100 emphasizes challenging real-world scenarios through a curated taxonomy of eight edge-case families, including face swaps, text-to-video stills, AI photo edits, avatar compositing, opaque-provenance images, and compressed research frames. Our evaluation shows that commercial APIs achieve the strongest median performance, followed by vision LLMs and open-source detectors. However, individual open-source models remain competitive with the best vision LLMs. More importantly, we identify a consistent divergence between ranking ability (ROC-AUC) and operating-point quality (MCC), demonstrating that strong score discrimination does not necessarily produce reliable default-threshold decisions. This metric disagreement, rather than any single leaderboard ranking, is the central finding of the benchmark. We release the complete evaluation framework and benchmark results to support reproducible future research. The source code and data are available at: https://github.com/sharayu-20/vendorbench-100