The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Baha Rababah, Cuneyt Gurcan Akcora, Carson K. Leung
• Venue: arXiv: cs.AI
• Published: 2026-07-09
• Collected: 2026-07-10
• URL: http://arxiv.org/abs/2607.08734v1
• PDF: https://arxiv.org/pdf/2607.08734v1
• DOI: UNKNOWN

Plain-Language Brief

This research paper introduces a new evaluation metric called 'correctness agreement' for large language models that undergo post-training quantization, a method used to deploy these models in resource-constrained settings. The authors argue that the current evaluation methods based on accuracy and perplexity do not capture behavioral changes caused by quantization. They find that even when task performance seems preserved under moderate quantization, there are behavioral divergences between the base model and its quantized versions. To explain this, they analyze quantization as a structural operator on attention weights and provide statistical and distributional measures of layer-wise distortions.

What the Document Says

• The paper discusses post-training quantization for large language models in resource-constrained settings.
• Evaluation is currently based on accuracy and perplexity, but the authors claim these metrics fail to capture behavioral changes caused by quantization.
• A new metric called 'correctness agreement' is introduced which measures overlap in correct predictions between a base model and its quantized variants, independent of absolute accuracy.
• Behavioral divergence emerges under moderate quantization even when task performance appears preserved.
• Quantization is analyzed as a structural operator on attention weights.
• The authors provide statistical and distributional measures of layer-wise distortions.
• They find non-linear breakpoints at low bit-widths.

Key Concepts Mentioned

• post-training quantization
• large language models
• resource-constrained settings
• correctness agreement
• accuracy
• perplexity
• attention weights
• bit-widths

Claims Made by the Document

• Evaluation based on accuracy and perplexity fails to capture behavioral changes caused by quantization.
• Behavioral divergence emerges under moderate quantization even when task performance appears preserved.

Evidence Used

• Results showing behavioral divergence under moderate quantization.
• Statistical and distributional measures of layer-wise distortions.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Post-training quantization is widely used to deploy large language models in resource-constrained settings, yet its evaluation relies almost exclusively on accuracy and perplexity. We show that these metrics fail to capture behavioral changes induced by quantization. We introduce correctness agreement, a decision-level metric that measures overlap in correct predictions between a base model and its quantized variants, independent of absolute accuracy. Across multiple models and quantization schemes from 8-bit to 2-bit, we find that behavioral divergence emerges under moderate quantization even when task performance appears preserved. To explain this effect, we analyze quantization as a structural operator on attention weights and quantify layer-wise distortions using statistical and distributional measures. Our results reveal non-linear breakpoints at low bit-widths and show that query and key projections are consistently more sensitive than value and output projections. These findings expose an illusion of equivalence between base and quantized models and motivate behavioral evaluation beyond conventional performance metrics.