Psychological Competence as a Missing Dimension in AI Evaluation

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Marcos Economides, Paul M. Sacher, Samuel Salzer, Alexis Michelle Abellar, Fendi Tsim, Antoine Ferrère
• Venue: arXiv: cs.AI
• Published: 2026-07-09
• Collected: 2026-07-10
• URL: http://arxiv.org/abs/2607.08285v1
• PDF: https://arxiv.org/pdf/2607.08285v1
• DOI: UNKNOWN

Plain-Language Brief

The paper discusses the need to consider psychological competence as an important aspect in evaluating AI systems that interact directly with humans, particularly in roles like advisors, coaches, tutors, and companions. Psychological competence refers to a human-facing AI system's ability to support user cognition, emotional interpretation, and behavioral decision-making appropriately for the user, context, and purpose of the interaction.

What the Document Says

• AI evaluation frameworks primarily focus on technical performance aspects like accuracy, robustness, reasoning ability, and policy compliance.
• Human-facing AI systems are used as advisors, coaches, tutors, and companions, and their responses can influence user's reasoning, emotions, beliefs, trust, and decisions.
• The relevant unit of evaluation is the human-AI interaction.
• Psychological competence is introduced as a missing dimension in AI evaluation, defined as the capacity of a human-facing AI system to support user cognition, emotional interpretation, and behavioral decision-making appropriately for the user, context, and purpose of the interaction.
• This includes interaction properties such as framing, tone, perceived authority, responsiveness, uncertainty handling, and conversational guidance.
• Existing evaluation approaches capture parts but not all of this problem.
• The paper outlines a conceptual framework for psychological competence and its core domains.
• It does not propose a specific benchmark but describes how psychological competence may be assessed through scenario-based probes, structured human evaluation, and model-assisted evaluation methods.
• Psychological competence should be a core consideration for model providers, deploying organizations, researchers, and regulators concerned with the real-world effects of human-facing AI systems.

Key Concepts Mentioned

• AI
• human-AI interaction
• psychological competence
• technical performance
• evaluation framework

Claims Made by the Document

• Current evaluation frameworks are not sufficient for human-facing AI systems.
• Psychological competence is a missing dimension in AI evaluation.
• Existing evaluation approaches capture parts but not all of psychological effects in human-AI interaction.
• Psychological competence should be a core consideration for model providers, deploying organizations, researchers, and regulators.

Evidence Used

• Human-facing AI systems are used as advisors, coaches, tutors, and companions.
• The paper's definition of psychological competence.

Methods / Approach

• Scenario-based probes
• Structured human evaluation
• Model-assisted evaluation methods

Limitations Stated by the Authors

Does not propose a specific benchmark for psychological competence assessment.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The paper does not discuss the technical feasibility or practicality of implementing the proposed psychological competence assessment methods.

Open Questions

• What specific benchmarks could be developed for psychological competence assessment?
• How can the technical feasibility and practicality of the proposed psychological competence assessment methods be evaluated?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper focuses on introducing the concept of psychological competence and providing a general framework for its assessment, but does not delve into specific details such as how to develop benchmarks or evaluate feasibility.

Raw Abstract / Excerpt

Current AI evaluation frameworks focus primarily on technical performance, including accuracy, robustness, reasoning ability, and policy compliance. These measures remain essential, but they are not sufficient for systems that interact directly with users through natural language. Human-facing AI systems are increasingly used as advisors, coaches, tutors, and companions. In these roles, their responses can shape how users reason, interpret emotions, form beliefs, calibrate trust, and make decisions. The relevant unit of evaluation is therefore not only the model, but the human-AI interaction. This paper introduces psychological competence as a missing dimension in AI evaluation. We define psychological competence as the capacity of a human-facing AI system to support user cognition, emotional interpretation, and behavioral decision-making in ways that are appropriate to the user, context, and purpose of the interaction. This includes interaction properties such as framing, tone, perceived authority, responsiveness, uncertainty handling, and conversational guidance. Existing evaluation approaches capture parts of this problem but rarely assess these psychological effects directly. Drawing on behavioral science and human-AI interaction research, we outline a conceptual framework for psychological competence and its core domains. Rather than proposing a specific benchmark, we define the construct, clarify its boundaries, and describe how it may be assessed through scenario-based probes, structured human evaluation, and model-assisted evaluation methods. We argue that psychological competence should become a core consideration for model providers, deploying organizations, researchers, and regulators concerned with the real-world effects of human-facing AI systems.