VICBench: A Multi-Language Benchmark for Code Vulnerability Detection

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Jin Lu, Xuening Han, Yang Zhong, Lin Tan, Kevin Luo, Andrew Gacek, Neha Rungta
• Venue: arXiv: cs.CR
• Published: 2026-08-12
• Collected: 2026-08-13
• URL: http://arxiv.org/abs/2608.12246v1
• PDF: https://arxiv.org/pdf/2608.12246v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

Researchers have created a benchmark dataset called VICBench for evaluating code vulnerability detection tools. The dataset contains 100 verified vulnerabilities across 88 projects in Python, Java, and C++, covering 48 types of common security weaknesses (CWE). The VICBench dataset includes complex real-world vulnerability fixes, averaging 38.6 lines, and corresponding vulnerable commits, averaging 252.5 lines. State-of-the-art algorithms performed poorly on this dataset, with F1 scores of only 33.3%-40.1%, indicating that existing approaches still require significant manual effort.

What the Document Says

• VICBench is a benchmark dataset for evaluating code vulnerability detection tools.
• The dataset contains 100 verified vulnerabilities across 88 projects in Python, Java, and C++.
• It covers 48 types of common security weaknesses (CWE).
• VICBench includes complex real-world vulnerability fixes averaging 38.6 lines and corresponding vulnerable commits of 252.5 lines.
• State-of-the-art algorithms V-SZZ and LLM4SZZ achieved F1 scores of 33.3%-40.1% on the VICBench dataset, indicating significant manual effort is still required.

Key Concepts Mentioned

• VICBench
• code vulnerability detection tools
• vulnerabilities
• CVEs
• CWE types
• complex real-world vulnerability fixes

Claims Made by the Document

• VICBench enables robust evaluation of vulnerability detection approaches.

Evidence Used

• The dataset was created through a dual annotation by human experts and an agentic workflow.
• The evaluation results for state-of-the-art algorithms were provided.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Evaluating security vulnerability detection tools requires benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first introduce vulnerabilities into codebases. VICs are essential for determining the full range of vulnerable software versions. Existing vulnerability datasets suffer from limited programming language coverage, restricted patch complexity, and narrow project scope. Through our dual annotation by human experts and an agentic workflow, we create a benchmark - VICBench - of 100 verified VICs for 100 CVEs across 88 projects in Python, Java, and C++, covering 48 CWE types. VICBench features complex real-world vulnerability fixes averaging 38.6 lines and corresponding VICs of 252.5 lines - significantly larger than prior work. Our evaluation shows that state-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1, confirming that using existing approaches still entails significant manual effort. VICBench enables robust evaluation of vulnerability detection approaches.