Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models

Source

arXiv

arxiv.org/abs/2606.25519v1

Read the source

Source

Plain-Language Brief

A study finds that using low-bit quantization to reduce the cost of large language models can lead to longer chains of thought, which offsets the expected per-token speedup in reasoning tasks. The study introduces a new metric called CoT Token Inflation Ratio to measure this effect and explores potential solutions like prompting, decoding-time sampling, and quantization-aware training.

What the Document Says

  • The text discusses the use of quantization to reduce the inference cost of large language models.
  • It mentions that low-bit post-training quantization can introduce a hidden test-time compute cost, causing quantized reasoning models to generate longer chains of thought even when they still answer correctly.
  • Across various benchmarks like mathematical reasoning, code generation, scientific question answering, and agentic tool-use, the study finds that INT4/INT3 quantization can preserve accuracy but increase reasoning-token usage.
  • The CoT Token Inflation Ratio is introduced as a metric to compare reasoning length between quantized and full-precision models across all evaluation benchmarks.
  • The text also notes behavioral changes in the reasoning trace, such as more intermediate steps and greater semantic repetition with token inflation.
  • These changes translate into measurable end-to-end real-world serving penalties.
  • The study evaluates mitigation strategies like prompting, decoding-time sampling, and quantization-aware training to address the issue of token inflation.

Key Concepts Mentioned

  • quantization
  • large language models
  • inference cost
  • low-bit post-training quantization
  • hidden compute cost
  • reasoning models
  • token inflation
  • CoT Token Inflation Ratio
  • mathematical reasoning
  • code generation
  • scientific question answering
  • agentic tool-use
  • prompting
  • decoding-time sampling
  • quantization-aware training

Claims Made by the Document

  • Low-bit post-training quantization can introduce a hidden test-time compute cost.
  • INT4/INT3 quantization can preserve accuracy but increase reasoning-token usage.
  • Token inflation is accompanied by behavioral changes in the reasoning trace.

Evidence Used

  • The text discusses the effect of low-bit post-training quantization on reasoning models and the resulting longer chains of thought.
  • Across various benchmarks, the study provides evidence of increased reasoning-token usage with INT4/INT3 quantization.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Quantization is widely used to reduce the inference cost of large language models, but its effect on reasoning models is not fully captured by final-answer accuracy or per-token latency. We show that low-bit post-training quantization can introduce a hidden test-time compute cost: quantized reasoning models often generate longer chains of thought even when they still answer correctly. Across mathematical reasoning, code generation, scientific question answering, and agentic tool-use benchmarks, we find that INT4/INT3 quantization can preserve accuracy but increase reasoning-token usage, offsetting the expected per-token speedup. To measure this effect, we introduce the CoT Token Inflation Ratio, which compares reasoning length between quantized and full-precision models averaged across all evaluation benchmarks. We further show that token inflation is accompanied by behavioral changes in the reasoning trace, including more intermediate steps and greater semantic repetition. These changes translate into measurable end-to-end real-world serving penalties. Finally, we evaluate mitigation strategies and find that prompting and decoding-time sampling offer inconsistent accuracy-length trade-offs, while quantization-aware training shows more promise in reducing both accuracy degradation and token inflation. Our results suggest that reasoning-token usage should be reported alongside accuracy when evaluating quantized reasoning models.