How Do Large Language Models Judge Social Attraction? Evidence from Theory-Grounded Persona Ratings Across Multiple LLMs and Humans

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Hasan Mahmud, Khawaja Abaid Ullah, Mohammad Javad Khojasteh, Jamison Heard, Prabu David
• Venue: arXiv: cs.CL
• Published: 2026-08-10
• Collected: 2026-08-11
• URL: http://arxiv.org/abs/2608.09717v1
• PDF: https://arxiv.org/pdf/2608.09717v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

This research investigates how large language models (LLMs) evaluate social attraction by comparing their judgments to human's from theory-grounded persona ratings. The study found that while some LLMs tended to give higher or lower ratings overall, they showed stability across runs and had high agreement in relative profile ordering. No significant effects of gender presentation were found in either the LLMs or humans evaluations.

What the Document Says

• The paper discusses the ability of large language models (LLMs) to evaluate social attraction using theory-grounded persona profiles.
• In Study 1, 34 LLMs rated 12 personas across three runs and showed strong stability, consistent ordering, and high agreement in relative profile ordering.
• Study 2 examined sensitivity to gender presentation with six matched name-and-pronoun profile pairs and a separate pronoun-only test. No significant effects were found.
• In Study 3, 198 human participants evaluated the same profiles as Study 2. Their ratings reproduced the three-tier structure and followed a consistent ordering with the LLMs.

Key Concepts Mentioned

• Large Language Models
• Social Attraction
• Persona Profiles
• Study
• Human Participants

Claims Made by the Document

• Some LLMs tend to give higher or lower ratings overall but show stability across runs and high agreement in relative profile ordering.
• No significant effects of gender presentation were found in either the LLMs or humans evaluations.

Evidence Used

• Stability, consistency, and agreement in LLM's ratings across runs
• Lack of significant effects in gender presentation analysis

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large language models (LLMs) are increasingly used to perform subjective evaluations traditionally made by humans, yet their validity as social judges remains unclear. This paper examines whether LLMs can assess social attraction from theory-grounded persona profiles constructed from ten psychological and relational constructs and organized into three tiers: socially attractive, socially mixed, and socially unattractive. We examine LLM ratings in two studies and compare them with human judgments in a third study. In Study 1, 34 LLMs rated 12 profiles across three repeated runs. Although some models tended to give higher or lower ratings overall, they showed strong stability across runs, consistent three-tier ordering, and high agreement in relative profile ordering. Study 2 examined sensitivity to gender presentation using six matched name-and-pronoun profile pairs and a separate pronoun-only test with a gender-neutral name, finding no significant effects in either analysis. In Study 3, 198 human participants evaluated the six matched profiles from Study 2. Their ratings reproduced the three-tier structure and followed a profile ordering consistent with that of the LLMs. However, LLMs rated attractive profiles more positively and unattractive profiles more negatively than humans, while neither group showed a significant overall effect of gender presentation.