VICBench: A Multi-Language Benchmark for Code Vulnerability Detection

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Jin Lu, Xuening Han, Yang Zhong, Lin Tan, Kevin Luo, Andrew Gacek, Neha Rungta
• Venue: arXiv: cs.CR
• Published: 2026-08-12
• Collected: 2026-08-13
• URL: http://arxiv.org/abs/2608.12246v1
• PDF: https://arxiv.org/pdf/2608.12246v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

The paper introduces VICBench, a multi-language benchmark for code vulnerability detection. It was created by human experts and an agentic workflow, featuring 100 verified vulnerabilities across 88 projects in Python, Java, and C++, covering 48 CWE types. The VICs are larger than previous work, averaging 252.5 lines for the vulnerable commit and 38.6 lines for the fix. State-of-the-art algorithms perform poorly, achieving only 33.3%-40.1% F1 score.

What the Document Says

• VICBench is a benchmark for code vulnerability detection created by human experts and an agentic workflow.
• It includes 100 verified VICs across 88 projects in Python, Java, and C++, covering 48 CWE types.
• The average size of the vulnerable commit (VIC) is 252.5 lines, and the average size of the fix is 38.6 lines.
• State-of-the-art algorithms V-SZZ and LLM4SZZ achieve F1 scores of 33.3%-40.1%.
• VICBench allows for robust evaluation of vulnerability detection approaches.

Key Concepts Mentioned

• code vulnerability detection
• benchmark
• VICBench
• human experts
• agentic workflow
• Python
• Java
• C++
• CVEs
• CWE types
• vulnerable commit (VIC)
• fix
• state-of-the-art algorithms
• F1 score

Claims Made by the Document

• VICBench provides a robust evaluation of vulnerability detection approaches.
• State-of-the-art algorithms perform poorly on VICBench.

Evidence Used

• The paper describes the creation and characteristics of VICBench.
• The F1 scores for state-of-the-art algorithms are presented in the paper.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Evaluating security vulnerability detection tools requires benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first introduce vulnerabilities into codebases. VICs are essential for determining the full range of vulnerable software versions. Existing vulnerability datasets suffer from limited programming language coverage, restricted patch complexity, and narrow project scope. Through our dual annotation by human experts and an agentic workflow, we create a benchmark - VICBench - of 100 verified VICs for 100 CVEs across 88 projects in Python, Java, and C++, covering 48 CWE types. VICBench features complex real-world vulnerability fixes averaging 38.6 lines and corresponding VICs of 252.5 lines - significantly larger than prior work. Our evaluation shows that state-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1, confirming that using existing approaches still entails significant manual effort. VICBench enables robust evaluation of vulnerability detection approaches.