AISPA: User-Centric System Prompt Auditing for Large Language Model Applications

Source

Source name: arXiv
Source kind: paper
Authors / organization: Xiangning Lin, Shenzhe Zhu, Shu Yang, Zhenyu Zhang, Haoqian Zhang, Yipeng Zhao, Chengxuan Qian, Tianwei Wang, Ziheng Zhang, Zhenlong Yuan, Dingcheng Wang, Juncheng Wu, Yuan Si, Jiaxin Liu, Baolong Bi, Robert Mahari, Tobin South, Dazza Greenwood, Zexue He, Rishi Bommasani, Sophia Kazinnik, Andreas Haupt, Samuele Marro, Erik Brynjolfsson, Alex Pentland, Jiaxin Pei
Venue: arXiv: cs.AI
Published: 2026-07-30
Collected: 2026-07-31
URL: http://arxiv.org/abs/2607.28617v1
PDF: https://arxiv.org/pdf/2607.28617v1
DOI: UNKNOWN

Plain-Language Brief

A new framework called AISPA is introduced to audit system prompts in AI systems, which are instructions used by developers to control the behavior of AI applications. The framework examines specific parts of a system prompt and evaluates them across eight user-relevant dimensions. An audit of 3249 instructions from 88 commercial AI products shows that while protective instructions are widely adopted, problematic ones remain pervasive. The findings highlight the need for greater transparency, standardization, and independent oversight in system prompts in commercial AI products.

What the Document Says

AISPA is a framework for auditing system prompts in AI systems.
The framework examines specific parts of a system prompt and evaluates them along eight dimensions that matter to users.
An audit was conducted on 3249 instructions from 88 commercial AI products.
The audit classified each instruction as either protective (of users) or problematic.
Findings show that system prompt design varies significantly across products and developers.
Protective instructions are widely adopted but have shallow scope, with only 24% of products covering all eight dimensions of the AISPA taxonomy.
System prompts have grown longer and more protective over time.
Despite progress, problematic instructions remain pervasive, with roughly 40% of products containing at least one instruction that works against user interests.

Key Concepts Mentioned

AISPA
system prompts
AI systems
commercial AI products
protective instructions
problematic instructions

Claims Made by the Document

AISPA is a framework for auditing system prompts.
The audit shows that system prompt design varies significantly across products and developers.
Protective instructions are widely adopted but have shallow scope.
System prompts have grown longer and more protective over time.
Problematic instructions remain pervasive in commercial AI products.

Evidence Used

AISPA, system prompt auditing framework
3249 instructions from 88 commercial AI products

Methods / Approach

System prompt audit using the AISPA framework

Limitations Stated by the Authors

Not stated in source

Limitations Noted by the Agent

Agent interpretation — not from the source.

The findings are based on an audit of a specific number of system prompts from commercial AI products. It is unknown if the results can be generalized to all AI systems.

Open Questions

What are the long-term implications of these findings for the development and deployment of AI systems?
How can greater transparency, standardization, and independent oversight be achieved in system prompts in commercial AI products?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper discusses the need for transparency, standardization, and independent oversight but does not provide specific solutions or strategies.

Raw Abstract / Excerpt

System prompts are instructions configured by developers to govern the behaviors of foundation models in AI applications. They are used throughout commercial AI products, but are rarely disclosed to the public or regulators, creating a serious trust and accountability gap in the wide deployment of AI systems. In this paper, we introduce Artificial Intelligence System Prompt Assurance (AISPA), a user-centric framework for systematically auditing system prompts in AI systems. AISPA examines specific parts of a system prompt and evaluates them along eight dimensions that matter to users. We then use this framework to review 3,249 instructions from system prompts in 88 commercial AI products, classifying each instruction as either protective (of users) or problematic. Our audit surfaces four core findings. First, system prompt design varies substantially across products and developers, with some organizations averaging over 60 protective instructions per product while others average fewer than 5. Second, protective instructions are widely adopted but shallow in scope: 98.9% of products contain at least one, yet only 24% cover all eight dimensions of the AISPA taxonomy. Third, system prompts have grown steadily longer and more protective of users, suggesting that user protection is becoming a more visible concern in commercial prompt design. Fourth, despite this progress, problematic instructions remain pervasive: roughly 40% of products contain at least one instruction that works against user interests, and protective and problematic instructions frequently coexist within the same prompt. Our findings highlight the need for greater transparency, standardization, and independent oversight for system prompts in commercial AI products.