Poli-Bias: Understanding and Measuring Large Language Model Biases in International Political Conflicts

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Massi-Nissa Abboud, Aladin Djuhera, Elena Cabrio, Holger Boche
• Venue: arXiv: cs.AI
• Published: 2026-08-06
• Collected: 2026-08-07
• URL: http://arxiv.org/abs/2608.06123v1
• PDF: https://arxiv.org/pdf/2608.06123v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers introduce Poli-Bias, a method to measure political bias in large language models (LLMs) by comparing responses to paired prompts where country identities are swapped across various geopolitical relationships, legal violations, and reasoning tasks. The framework breaks down response disparities into five interpretable dimensions, revealing where unequal treatment occurs.

What the Document Says

• Poli-Bias is a counterfactual framework for measuring bias in LLMs
• The framework compares responses to paired prompts with swapped country identities
• It decomposes response disparities into five interpretable dimensions
• Across 13 LLMs, the researchers find that country identities and user affiliations can affect how equivalent actions are described, evaluated, and defended under international law.

Key Concepts Mentioned

• Poli-Bias
• Large Language Models (LLMs)
• Counterfactual framework
• Country identities
• User affiliations
• International law

Claims Made by the Document

• The framework can measure bias in LLMs
• Country identities and user affiliations can affect how equivalent actions are described, evaluated, and defended under international law

Evidence Used

• Across 13 contemporary LLMs spanning diverse model families and sizes, we find that country identities and user affiliations can systematically affect how equivalent actions are described, evaluated, and defended under international law.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Measuring political bias in large language models (LLMs) remains challenging as it can manifest through subtle differences in framing, argumentation, and legal reasoning that are difficult to capture with a single metric. In this work, we introduce Poli-Bias, a counterfactual framework for measuring whether LLMs treat legally equivalent conflict scenarios differently depending on the countries involved. Poli-Bias compares responses to paired prompts in which country identities are systematically swapped across diverse geopolitical relationships, legal violations, and reasoning tasks. Rather than reducing bias to a single judgment, our framework decomposes response disparities into five interpretable dimensions, revealing how and where unequal treatment manifests. Across 13 contemporary LLMs spanning diverse model families and sizes, we find that country identities and user affiliations can systematically affect how equivalent actions are described, evaluated, and defended under international law. Our results thus establish Poli-Bias as a fine-grained framework for auditing political even-handedness and sycophancy in LLMs.