Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

When Specifications Conflict: A Symmetry-Based Framework for Measuring LLM Preferences

Source

arXiv

arxiv.org/abs/2607.28384v1

Read the source

Source

Plain-Language Brief

The researchers propose a framework to analyze how large language models (LLMs) handle conflicting specifications when integrating multiple sources of information. They compare four types of representation: pure natural language, formal language, naturalized formal language, and input-output examples, using an executable mathematical benchmark with 550 conflict instances spanning 11 function families.

What the Document Says

  • The study focuses on analyzing how LLMs handle conflicting specifications in integrating multiple sources of information.
  • They propose a framework for observing and analyzing model choices between competing specifications with explicit conflicts.
  • A symmetry-based design is used to reduce confounding factors, allowing for systematic comparison of preferences across representation types.
  • The evaluation uses an executable mathematical benchmark with 550 conflict instances spanning 11 function families and four representation types: pure natural language, formal language, naturalized formal language, and input-output examples.
  • Results show a consistent ordering of preference patterns for the representation types: Formal ≈ Naturalized Formal > Pure Natural Language > Input-Output Examples.

Key Concepts Mentioned

  • large language models (LLMs)
  • conflicting specifications
  • controlled experimental framework
  • symmetry-based design
  • preferences across representation types
  • evaluation
  • formal language
  • naturalized formal language
  • input-output examples

Claims Made by the Document

  • There is a lack of controllable and attributable methods for analyzing how models resolve conflicts between competing specifications.
  • The proposed framework allows model choices between competing specifications to be directly observed and analyzed.
  • A consistent ordering was found for the preference patterns across representation types: Formal ≈ Naturalized Formal > Pure Natural Language > Input-Output Examples.

Evidence Used

  • The text describes a framework and its purpose of studying model preferences under conflicting specifications.
  • The evaluation data and results are provided in the form of an executable mathematical benchmark with 550 conflict instances and four representation types.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large language models (LLMs) are increasingly required to integrate multiple sources of information that may be inconsistent or conflicting. However, there is still a lack of controllable and attributable methods for analyzing how models resolve conflicts between competing specifications. We propose a controlled experimental framework for studying model preferences under conflicting specifications. By constructing specifications with explicit conflicts, the framework enables model choices between competing specifications to be directly observed and analyzed. A symmetry-based design further reduces confounding factors, allowing preferences across representation types to be compared systematically. We evaluate the framework on an executable mathematical benchmark with 550 conflict instances spanning 11 function families, comparing four representation types: pure natural language, formal language, naturalized formal language, and input--output examples. Results show systematic preference patterns rather than random behavior, with a consistent ordering: $ \text{Formal} \approx \text{Naturalized Formal} > \text{Pure Natural Language} > \text{Input--Output Examples} $. Example effects further depend on model capability and function family. We extend the framework to heterogeneous specification conflicts in Boolean algebra, code generation, and the clinical domain, demonstrating its applicability across diverse tasks and specification forms. The framework provides a unified approach for measuring how LLMs resolve conflicts between competing sources of information.