Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detection

Source

arXiv

arxiv.org/abs/2606.30587v1

Read the source

Source

Plain-Language Brief

In this research, the authors investigate how Large Language Models (LLMs) used for automated vulnerability detection in code are influenced by cognitive biases similar to those that affect human judgment. They present a controlled framework and evaluate eight LLMs across three programming languages, revealing that these models are susceptible to cognitive heuristics like the halo effect, framing effect, and anchoring effect. The study shows that vulnerabilities requiring semantic reasoning for detection are more susceptible to these biases than those identifiable through pattern matching, and that models can change their verdict based on cognitive conditions without accurately identifying the actual vulnerability.

What the Document Says

  • The paper investigates LLM-driven code vulnerability detection and its susceptibility to cognitive heuristics that influence human judgment.
  • A controlled framework is introduced with a fixed code and varying context to trigger cognitive heuristics: halo effect, framing effect, and anchoring effect.
  • Eight different LLMs are evaluated across three programming languages in this framework, performing both quantitative and code-level analyses.
  • The findings demonstrate that all evaluated models are susceptible to these cognitive heuristics. Susceptibility is highest for framing (33.2%), followed by anchoring (23.5%) and halo (18.4%).
  • Code-level analysis indicates that semantic reasoning-based vulnerabilities are more susceptible to cognitive heuristics than those identifiable through pattern matching.
  • Models often change their verdict from safe to vulnerable based on cognitive conditions without accurately identifying the actual vulnerability.
  • A proof-of-concept black-box cognitive attack is demonstrated, capable of suppressing up to 97% of previously detected vulnerabilities.

Key Concepts Mentioned

  • Large Language Models (LLMs)
  • Code vulnerability detection
  • Cognitive heuristics
  • Halo effect
  • Framing effect
  • Anchoring effect

Claims Made by the Document

  • LLMs are susceptible to cognitive heuristics that bias human judgment.
  • The halo effect, framing effect, and anchoring effect affect LLM-driven code vulnerability detection.
  • Vulnerabilities requiring semantic reasoning for detection are more susceptible to cognitive heuristics than those identifiable through pattern matching.

Evidence Used

  • Evaluation of eight LLMs across three programming languages in a controlled framework that varies the context to trigger cognitive heuristics.
  • Quantitative and code-level analyses demonstrating the models' susceptibility to cognitive heuristics.
  • Code-level analysis revealing higher susceptibility for semantic reasoning-based vulnerabilities.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Researchers and practitioners increasingly apply Large Language Models (LLMs) for automated vulnerability detection. Recent work has shown that LLMs are susceptible to the same cognitive heuristics that bias human judgment. Yet, no work has investigated whether these heuristics affect a model's assessment of code vulnerabilities. In this paper, we present the first systematic exploration of cognitive heuristics in LLM-driven code vulnerability detection. We introduce a controlled framework that holds the code fixed and only varies the surrounding context to trigger three cognitive heuristics: the halo effect through author attribution, the framing effect through task objectives and consequences, and the anchoring effect through prior analysis results. Within this framework, we evaluate eight LLMs across three programming languages and perform both quantitative and code-level analyses. Our findings demonstrate that all evaluated models are susceptible to these heuristics. Cross-model average susceptibility is highest for framing at 33.2%, followed by anchoring at 23.5% and halo at 18.4%. Code-level analysis reveals that vulnerabilities that require semantic reasoning for detection are more susceptible to cognitive heuristics than those identifiable through pattern matching. Furthermore, models often change their verdict from safe to vulnerable based on the cognitive condition, without accurately identifying the actual vulnerability. To highlight the practical impact, we demonstrate a proof-of-concept black-box cognitive attack that can suppress up to 97% of previously detected vulnerabilities. These findings indicate that cognitive susceptibility is a consistent and exploitable property of LLM-based vulnerability detection.