Rethinking Penetration Testing for AI-Enabled Systems: From Resource Compromise to Behavioral Objective Violation

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Mohammad Allahbakhsh, Mohammad Hassan Bahari, Moslem Attar-Raouf
• Venue: arXiv: cs.CR
• Published: 2026-07-15
• Collected: 2026-07-16
• URL: http://arxiv.org/abs/2607.14006v1
• PDF: https://arxiv.org/pdf/2607.14006v1
• DOI: UNKNOWN

Plain-Language Brief

The paper proposes a new approach to penetration testing for AI-enabled systems, focusing on evaluating whether adversaries can manipulate system behavior without directly compromising the underlying infrastructure. The authors define AI-enabled penetration as inducing AI-governed behavior that violates operational objectives under an explicit threat model.

What the Document Says

• Penetration testing is traditionally used to evaluate the compromise of software, infrastructure, configurations, or operational controls.
• For AI-enabled systems, penetration testing needs to be rethought as objective-driven behavioral evaluation.
• AI-enabled systems are defined as those where learned models materially influence behavior affecting operational outcomes.
• AI-enabled penetration is defined as the feasible induction of AI-governed behavior that violates one or more operational objectives under an explicit threat model.
• The proposed approach includes identifying operational objectives, mapping AI-governed behavior, analyzing adversarial influence surfaces, defining behavioral failure criteria, executing scenario-based tests, and reporting evidence.
• A running example is provided involving an AI-enabled security operations center assistant.
• The paper provides a technical framework for evaluating adversarial success in deployed AI-enabled systems.

Key Concepts Mentioned

• Penetration testing
• AI-enabled systems
• AI-governed behavior
• Operational objectives
• Adversarial influence surfaces
• Behavioral failure criteria
• Scenario-based tests

Claims Made by the Document

• Traditional penetration testing is not sufficient for AI-enabled systems.
• AI-enabled penetration extends conventional penetration testing to include indirect adversarial pathways.

Evidence Used

• The paper presents a new approach to penetration testing specifically tailored for AI-enabled systems.

Methods / Approach

• Defining a technical framework for evaluating adversarial success in deployed AI-enabled systems.

Limitations Stated by the Authors

The proposed methodology may not be applicable to all types of AI-enabled systems or threat scenarios.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The provided text does not specify the extent of the applicability of the proposed methodology in real-world situations.

Open Questions

• What are some potential real-world applications of this redefined penetration testing approach?
• How effective is this approach in identifying and mitigating various types of AI-related security threats?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper presents a theoretical framework for a new approach to penetration testing, but the practical effectiveness and applicability in various scenarios are not fully explored.

Raw Abstract / Excerpt

Penetration testing traditionally evaluates whether adversaries can exploit weaknesses in software, infrastructure, configurations, or operational controls to achieve security-relevant compromise. This paradigm remains necessary for AI-enabled systems, but it is no longer sufficient. In such systems, adversaries may influence prompts, retrieved content, sensor inputs, training data, memory, tools, or human-AI interaction loops to alter system behavior without directly compromising the underlying infrastructure. This paper reframes penetration testing for AI-enabled systems as objective-driven behavioral evaluation. We define an AI-enabled system as one in which learned models materially influence behavior affecting operational outcomes, and we define AI-enabled penetration as the feasible induction of AI-governed behavior that violates one or more operational objectives under an explicit threat model. This definition preserves conventional penetration testing while extending it to adversarial pathways such as prompt injection, indirect prompt injection, data poisoning, sensor manipulation, retrieval poisoning, tool misuse, and agentic misalignment. We further propose a testing workflow that identifies operational objectives, maps AI-governed behavior, analyzes adversarial influence surfaces, defines behavioral failure criteria, executes scenario-based tests, and reports evidence linking adversarial action to objective violation. A running example involving an AI-enabled security operations center assistant illustrates how penetration may occur through behavioral influence rather than infrastructure compromise. Together, the definitions, workflow, and example provide a technical framework for evaluating adversarial success in deployed AI-enabled systems.