Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Can LLMs Rank? A Tale of Triads and Triage

Source

arXiv

arxiv.org/abs/2606.30412v1

Read the source

Source

Plain-Language Brief

The authors discuss the use of Large Language Models (LLMs) in ranking people for scarce resources, such as housing or emergency services. They present two methods for assessing the consistency of LLM's judgments: one is a coefficient of consistency ($ζ$), which measures intra-run consistency, and another uses standard measures of distance between rankings, like Kendall's $τ$, to measure inter-run variability. The study demonstrates these methods on two high-stakes prioritization tasks: homelessness service allocation and emergency department triage, showing that different LLMs have varying performance profiles.

What the Document Says

  • LLMs are being used for decision making in areas like housing allocation and emergency services triage.
  • Ranking large groups is cognitively demanding and error-prone, so pairwise comparisons are aggregated into a total order.
  • The article discusses two methods to assess the consistency of LLM's judgments: coefficient of consistency ($ζ$) for intra-run consistency and standard measures of distance between rankings (like Kendall's $τ$) for inter-run variability.
  • The study demonstrates these methods on tasks involving homelessness service allocation and emergency department triage.
  • Three leading LLMs have different performance profiles across these two axes of consistency.

Key Concepts Mentioned

  • Large Language Models (LLMs)
  • Ranking
  • consistency
  • coefficient of consistency ($ζ$)
  • Kendall's $τ$
  • housing allocation
  • emergency services triage

Claims Made by the Document

  • LLMs are used for consequential decisions.
  • Pairwise comparisons are used to aggregate rankings.
  • Two methods exist for assessing the consistency of LLM's judgments.
  • The study demonstrates these methods on high-stakes prioritization tasks.

Evidence Used

  • From housing allocation for households experiencing homelessness to triage in emergency departments, LLMs are increasingly being considered as judges of consequential decisions that require ranking people for scarce resources.
  • We discuss two different ways of identifying consistency.
  • We show, in both theory and practice...

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

From housing allocation for households experiencing homelessness to triage in emergency departments, LLMs are increasingly being considered as judges of consequential decisions that require ranking people for scarce resources. Ranking large groups simultaneously is cognitively demanding and error-prone. A natural solution, drawing on decades of social choice theory, elicits pairwise comparisons and aggregates them into a total order. However, a fundamental question remains when LLMs serve as the pairwise judge: how can a practitioner tell, before committing to a ranking, whether the LLM's judgments are sufficiently consistent to trust the result? We discuss two different ways of identifying consistency. A classical diagnostic, the coefficient of consistency $ζ$, originally developed to measure judge reliability by counting circular triads in tournament graphs, provides a cheap, model-free measure of intra-run consistency. Various standard measures of distance between rankings, for example Kendall's $τ$, can measure inter-run variability. We show, in both theory and practice, that these measures are independently valuable, and advocate for using both to assess reliability of rankings. We demonstrate the practical importance of our results across two high-stakes prioritization tasks: homelessness service allocation and emergency department triage. Three different leading LLMs have considerably different performance profiles across these two axes of consistency. We provide guidelines for how practitioners could think about measuring and assessing consistency before committing to a model for ranking or prioritization.