Helpful or Harmful? Evaluating LLM-Assisted Vulnerability Patching via a Human Study

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Giulian Biolo, Michael Tezza, Yuanjun Gong, Fabio Massacci
• Venue: arXiv: cs.SE
• Published: 2026-06-24
• Collected: 2026-06-25
• URL: http://arxiv.org/abs/2606.25973v1
• PDF: https://arxiv.org/pdf/2606.25973v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers aim to evaluate the effectiveness of Large Language Models (LLMs) in assisting vulnerability patching compared to manual debugging by conducting an empirical experiment on human participants in real-world scenarios.

What the Document Says

• Software vulnerability remediation is a task requiring specialized security expertise.
• Large Language Models (LLMs) have potential in vulnerability detection, location, and repair tasks.
• Hypothesis: LLM-assistance may speed up patching but could also introduce errors leading to superficial repairs that might fail security validation.
• Objective: The experiment aims to present an empirical study comparing LLM-assisted vulnerability patching to manual debugging on human participants in real-world scenarios.
• Method: A controlled experiment using a Balanced Crossover design will be conducted with a WebApp for code execution and hidden Ghost Tests for verifying patch integrity.
• The experiment involves training and evaluation scenarios.
• Pilot Study: A small sample pilot experiment has been conducted to gather insights for the following study.

Key Concepts Mentioned

• Software vulnerability remediation
• Large Language Models (LLMs)
• human participants
• empirical experiment
• Balanced Crossover design
• WebApp
• Ghost Tests
• training scenarios
• evaluation scenarios

Claims Made by the Document

• LLM-assistance may speed up patching.
• LLM-assistance could introduce errors leading to superficial repairs that might fail security validation.

Evidence Used

• Not stated in source

Methods / Approach

• Controlled experiment using a Balanced Crossover design
• WebApp for code execution
• hidden Ghost Tests
• training scenarios
• evaluation scenarios

Limitations Stated by the Authors

Not stated in source

Limitations Noted by the Agent

Agent interpretation — not from the source.

The study only involves human participants, and the generalization of results to non-human agents is uncertain.

Open Questions

• What is the specific large language model being used for this study?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The text does not specify which Large Language Model (LLM) is being used in the experiment.

Raw Abstract / Excerpt

Software vulnerability remediation is a cognitively demanding task that requires specialized security expertise often lacking in general developers. In the meantime, Large Language Models (LLMs) assisted tools show potential in vulnerability detection, location, and repair tasks. [Hypothesis:] While LLM-assistance is hypothesized to accelerate patching, it also risks introducing hallucinations or insecure code, leading to a higher likelihood of generating superficial repairs that bypass the standard functionality checks but fail the security validation. [Objective:] We aim to present an empirical experiment, unveiling the capability of LLM-assisted vulnerability patching compared to manual debugging on human participants in real-world scenarios. [Method:] We plan to conduct a controlled experiment using a Balanced Crossover design. For that, we have developed a WebApp for code execution and integrated hidden Ghost Tests to verify patch integrity beyond visible functional requirements. The experiment involves training and evaluation scenarios. The remediation speed, remediation efficacy for both standard functionality tests and security tests, and participant perception will be evaluated. [Pilot Study:] A pilot experiment with a small sample of participants has been conducted, providing insights for the following study.