Breaking the weakest link to evade vision language models

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Ilan Zini, Boussad Addad, Katarzyna Kapusta
• Venue: arXiv: cs.AI
• Published: 2026-08-19
• Collected: 2026-08-20
• URL: http://arxiv.org/abs/2608.18938v1
• PDF: https://arxiv.org/pdf/2608.18938v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

The authors investigate the vulnerability of Vision Language Models (VLMs) to adversarial attacks, which can disrupt or force the models' interpretation of visual inputs. They propose a gradient-based attack method that reduces computational cost and resource requirements while maintaining effectiveness. The study shows that small, human-imperceptible perturbations can significantly alter the textual interpretations produced by several VLMs.

What the Document Says

• The paper discusses the insufficiently explored robustness of Vision Language Models (VLMs) against adversarial threats in multimodal AI systems.
• Two attack settings are studied: untargeted attacks and targeted attacks on VLMs.
• A gradient-based attack method is proposed, which optimizes exclusively on the vision encoder of the VLM to generate adversarial examples efficiently.

Key Concepts Mentioned

• Vision Language Models (VLMs)
• Adversarial attacks
• Untargeted attacks
• Targeted attacks
• Gradient-based attack method

Claims Made by the Document

• Small, human-imperceptible perturbations can substantially alter the textual interpretation produced by VLMs.

Evidence Used

• The findings from evaluating the approach on several open-source VLMs

Methods / Approach

• Gradient-based attack method

Limitations Stated by the Authors

• Not stated in source

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The study focuses on VLMs, and it is unknown if the conclusions apply to other types of AI systems.

Open Questions

• What are potential solutions for improving robustness and security mechanisms in multimodal AI systems?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The paper does not discuss possible solutions to improve the robustness of VLMs against adversarial attacks.

Raw Abstract / Excerpt

Vision Language Models (VLMs) have recently emerged as a critical component of multimodal AI systems, enabling joint reasoning over visual and textual inputs in real-world and safety-critical applications. Despite their growing deployment, the robustness of VLMs against adversarial threats remains insufficiently explored, particularly in the context of evasion attacks targeting multimodal alignment. In this work, we investigate the vulnerability of VLMs to adversarial perturbations applied to visual inputs and study two attack settings: untargeted attacks, where the goal is to disrupt the model's interpretation of the original image, and targeted attacks, where the adversary aims to force the model to generate a specific semantic description unrelated to the original image. To efficiently generate adversarial examples, we propose a gradient-based attack method that performs optimization exclusively on the vision encoder of the VLM rather than on the entire multimodal architecture. This design significantly reduces the computational cost and resource requirements of the attack while maintaining strong effectiveness. We evaluate our approach on several open-source VLMs, including Qwen2.5-VL, Granite-Vision, FastVLM, and Phi-3.5-Vision, and show that small, human-imperceptible perturbations can substantially alter the textual interpretation produced by the models. Our findings highlight the vulnerability of modern VLMs to adversarial manipulation and emphasize the need for improved robustness and security mechanisms in multimodal AI systems.