FIELD NOTES TOPIC
mitigation strategies
1 note, newest first.
Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models
A study finds that using low-bit quantization to reduce the cost of large language models can lead to longer chains of thought, which offsets the expected per-token speedup in reasoning tasks. The study introduces a…
