Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Collective cooperation without individual fidelity in LLM agents

Source

arXiv

arxiv.org/abs/2606.30454v1

Read the source

Source

Plain-Language Brief

Researchers tested large language models (LLMs) in a Prisoner's Dilemma experiment with human participants to evaluate if their behavior can be seen as a faithful representation of human decision-making. The LLM was able to replicate some macro-level features of cooperation dynamics, but it underestimated individual-level heterogeneity and generated different conditional cooperation patterns compared to humans.

What the Document Says

  • The study investigates the use of Large Language Models (LLMs) as agents in simulations of social systems.
  • The researchers compare LLM agents with human data from a large-scale networked Prisoner's Dilemma experiment.
  • The selected LLM reproduces some macro-level features of cooperation dynamics, but it does not uniformly match finer levels of behavior.
  • LLM populations underestimate individual-level heterogeneity and generate conditional cooperation patterns that differ from humans.
  • Adding a fraction of random agents improves some aspects of micro-level agreement but does not remove the mismatch in decision rules.
  • The findings indicate a macro--micro dissociation in LLM-based social agents, suggesting that validating LLMs as human surrogates requires comparisons across aggregate dynamics, individual heterogeneity, and context-dependent decision rules.

Key Concepts Mentioned

  • Large Language Models (LLMs)
  • Prisoner's Dilemma
  • human participants
  • cooperation dynamics
  • individual-level heterogeneity
  • decision rules

Claims Made by the Document

  • LLM agents can reproduce some macro-level features of cooperation dynamics in a Prisoner's Dilemma experiment.
  • LLM populations underestimate individual-level heterogeneity and generate conditional cooperation patterns that differ from humans.
  • Adding a fraction of random agents improves some aspects of micro-level agreement but does not remove the mismatch in decision rules.

Evidence Used

  • Results from a large-scale networked Prisoner's Dilemma experiment with human participants and LLM agents

Methods / Approach

  • Comparison of LLM agents with human data
  • Using the same interaction protocol, payoff structure, and network topologies in the experiments

Limitations Stated by the Authors

  • The study focuses on open-weight LLMs and does not explore other types of language models.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The agent may not be able to fully replicate human decision-making behavior in all scenarios, especially at the individual level.

Open Questions

  • How can we further improve LLMs to better mimic human decision-making at the individual level?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The study highlights a discrepancy between aggregate outcomes and individual behavior in LLM-based social agents.

Raw Abstract / Excerpt

Large language models (LLMs) are increasingly used as agents in simulations of social systems, yet it remains unclear when their behavior can be interpreted as a faithful proxy for human decision-making. Here we test LLM agents against a direct empirical benchmark: a large-scale networked Prisoner's Dilemma experiment with human participants. Using the same interaction protocol, payoff structure, and network topologies, we compare nine open-weight LLMs with the human data. The selected model reproduces several macro-level features of cooperation dynamics, including the early decline and later stabilization of cooperation. This aggregate agreement, however, does not extend uniformly to finer levels of behavior. LLM populations underestimate individual-level heterogeneity and generate conditional cooperation patterns that differ from those observed in humans. Adding a fraction of random agents improves some aspects of micro-level agreement, but does not remove the mismatch in decision rules. These findings reveal a macro--micro dissociation in LLM-based social agents: collective outcomes can appear human-like even when the underlying behavioral distributions and mechanisms are not. They suggest that validating LLM agents as human surrogates requires comparisons across aggregate dynamics, individual heterogeneity, and context-dependent decision rules, rather than outcome-level agreement alone.