Psychological Competence as a Missing Dimension in AI Evaluation

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Marcos Economides, Paul M. Sacher, Samuel Salzer, Alexis Michelle Abellar, Fendi Tsim, Antoine Ferrère
• Venue: arXiv: cs.AI
• Published: 2026-07-09
• Collected: 2026-07-10
• URL: http://arxiv.org/abs/2607.08285v1
• PDF: https://arxiv.org/pdf/2607.08285v1
• DOI: UNKNOWN

Plain-Language Brief

The text discusses a need to evaluate human-AI interaction beyond just technical performance, focusing on how AI systems can affect users' cognition, emotions, decision-making, and behavior. The proposed evaluation dimension is 'psychological competence', which encompasses factors like framing, tone, responsiveness, and uncertainty handling. The authors suggest various methods for assessing psychological competence, emphasizing its importance for model providers, deployers, researchers, and regulators.

What the Document Says

• The focus of AI evaluation is primarily on technical performance including accuracy, robustness, reasoning ability, and policy compliance.
• Human-facing AI systems are used as advisors, coaches, tutors, and companions, influencing users' cognition, emotions, decision-making, and behavior.
• The relevant unit of evaluation is the human-AI interaction.
• Psychological competence is defined as a capacity for a human-facing AI system to support user cognition, emotional interpretation, and behavioral decision-making appropriately for the user, context, and purpose of the interaction.
• Psychological competence includes interaction properties such as framing, tone, perceived authority, responsiveness, uncertainty handling, and conversational guidance.
• Existing evaluation approaches capture parts but not all of this problem.
• The authors outline a conceptual framework for psychological competence and its core domains.
• They propose scenario-based probes, structured human evaluation, and model-assisted evaluation methods to assess psychological competence.
• Psychological competence should be a core consideration for model providers, deployers, researchers, and regulators.

Key Concepts Mentioned

• AI
• human-AI interaction
• psychological competence
• evaluation
• cognition
• emotions
• decision-making
• behavior
• framing
• tone
• perceived authority
• responsiveness
• uncertaintly handling
• conversational guidance

Claims Made by the Document

• AI evaluation focuses primarily on technical performance.
• Human-facing AI systems can significantly influence users' cognition, emotions, decision-making, and behavior.
• Psychological competence is a missing dimension in current AI evaluation.
• Existing evaluation approaches capture only parts of the psychological effects problem.
• A conceptual framework for psychological competence has been outlined.
• Psychological competence should be a core consideration for various stakeholders.

Evidence Used

• The authors discuss how human-facing AI systems affect users' cognition, emotions, decision-making, and behavior.
• They describe the current focus of AI evaluation on technical performance.
• They define psychological competence and its components.
• They propose methods for assessing psychological competence.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Current AI evaluation frameworks focus primarily on technical performance, including accuracy, robustness, reasoning ability, and policy compliance. These measures remain essential, but they are not sufficient for systems that interact directly with users through natural language. Human-facing AI systems are increasingly used as advisors, coaches, tutors, and companions. In these roles, their responses can shape how users reason, interpret emotions, form beliefs, calibrate trust, and make decisions. The relevant unit of evaluation is therefore not only the model, but the human-AI interaction. This paper introduces psychological competence as a missing dimension in AI evaluation. We define psychological competence as the capacity of a human-facing AI system to support user cognition, emotional interpretation, and behavioral decision-making in ways that are appropriate to the user, context, and purpose of the interaction. This includes interaction properties such as framing, tone, perceived authority, responsiveness, uncertainty handling, and conversational guidance. Existing evaluation approaches capture parts of this problem but rarely assess these psychological effects directly. Drawing on behavioral science and human-AI interaction research, we outline a conceptual framework for psychological competence and its core domains. Rather than proposing a specific benchmark, we define the construct, clarify its boundaries, and describe how it may be assessed through scenario-based probes, structured human evaluation, and model-assisted evaluation methods. We argue that psychological competence should become a core consideration for model providers, deploying organizations, researchers, and regulators concerned with the real-world effects of human-facing AI systems.