Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection

Source

arXiv

arxiv.org/abs/2607.06254v1

Read the source

Source

Plain-Language Brief

The paper introduces VendorBench-100, a benchmark for evaluating deepfake image detection models across three paradigms: commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors. The benchmark uses a single corpus of 100 images, a unified output schema, and a common evaluation framework to facilitate comparison. The evaluation shows that commercial APIs generally perform best, followed by LLMs and open-source detectors. However, there's a discrepancy between the models' ranking ability (ROC-AUC) and their quality of decisions (MCC).

What the Document Says

  • The paper presents VendorBench-100, a benchmark for deepfake image detection that evaluates 36 representative models from three paradigms: commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors.
  • The benchmark uses a single adversarial 100-image corpus, a unified output schema, and a common evaluation framework for comparison.
  • The evaluation is primarily based on the Matthews correlation coefficient (MCC), with ROC-AUC reported as an additional measure.
  • The benchmark emphasizes challenging real-world scenarios through a curated taxonomy of eight edge-case families.
  • Commercial APIs, vision LLMs, and open-source detectors are ranked based on their performance. Commercial APIs achieve the strongest median performance followed by vision LLMs and open-source detectors.
  • However, there's a discrepancy between the models' ranking ability (ROC-AUC) and their quality of decisions (MCC).
  • The source code and data for VendorBench-100 are available at https://github.com/sharayu-20/vendorbench-100.

Key Concepts Mentioned

  • deepfake image detection
  • VendorBench-100
  • commercial APIs
  • zero-shot vision-language models (LLMs)
  • open-source detectors
  • Matthews correlation coefficient (MCC)
  • ROC-AUC

Claims Made by the Document

  • Commercial APIs achieve the strongest median performance.
  • There's a discrepancy between the models' ranking ability (ROC-AUC) and their quality of decisions (MCC).

Evidence Used

  • The evaluation shows that commercial APIs achieve the strongest median performance followed by vision LLMs and open-source detectors.
  • The paper demonstrates a consistent divergence between ranking ability (ROC-AUC) and operating-point quality (MCC), which is the central finding of the benchmark.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

  • It's unclear from the source whether there are any limitations in the selection of deepfake image detection models for evaluation.
  • The paper does not mention any potential biases or limitations in the curated taxonomy of edge-case families.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The agent understands that there could be other unmentioned limitations in the selection of deepfake image detection models and the creation of the benchmark.
  • The agent acknowledges the possibility of undisclosed biases or limitations in the curated taxonomy of edge-case families.

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • It's not stated in the source whether there are any specific deepfake image detection models being compared within each paradigm.
  • The paper does not provide information about the specific edge-case families within the curated taxonomy.

Raw Abstract / Excerpt

Deepfake image detection is currently served by three fundamentally different paradigms: commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors. Despite their widespread use, these paradigms are rarely evaluated under a common protocol, making direct comparison difficult. We introduce VendorBench-100, a cross-paradigm benchmark that evaluates 36 representative models using a single adversarial 100-image corpus, a unified output schema, and a common evaluation framework. To ensure reliable assessment under the corpus's intentional class imbalance, models are ranked primarily by the Matthews correlation coefficient (MCC), with ROC-AUC reported as a threshold-independent measure of ranking ability. Rather than maximizing dataset size, VendorBench-100 emphasizes challenging real-world scenarios through a curated taxonomy of eight edge-case families, including face swaps, text-to-video stills, AI photo edits, avatar compositing, opaque-provenance images, and compressed research frames. Our evaluation shows that commercial APIs achieve the strongest median performance, followed by vision LLMs and open-source detectors. However, individual open-source models remain competitive with the best vision LLMs. More importantly, we identify a consistent divergence between ranking ability (ROC-AUC) and operating-point quality (MCC), demonstrating that strong score discrimination does not necessarily produce reliable default-threshold decisions. This metric disagreement, rather than any single leaderboard ranking, is the central finding of the benchmark. We release the complete evaluation framework and benchmark results to support reproducible future research. The source code and data are available at: https://github.com/sharayu-20/vendorbench-100