A quantized LLM is not the same model — the "Illusion of Equivalency" in quantization effects

This work challenges a common assumption about LLM quantization: that if accuracy holds, the model is the same.

1Why post-training quantization became standard

Post-training quantization has become the standard technique for running LLMs on limited compute. But its quality evaluation has relied almost exclusively on aggregate metrics — accuracy and perplexity. The authors first show that these metrics fail to capture the behavioral changes quantization induces.

2Introducing correctness agreement

They introduce correctness agreement, a decision-level metric that measures the overlap in correct predictions between a base model and its quantized variants, independent of absolute accuracy. Even if aggregate accuracy is unchanged, the set of questions a model answers correctly can shift — and if it shifts, the two are not behaviorally the same model.

3What 8-bit to 2-bit revealed

Across multiple models and quantization schemes from 8-bit down to 2-bit, the study finds that behavioral divergence emerges under moderate quantization even when task performance appears preserved. To explain the effect, the authors analyze quantization as a structural operator on attention weights and quantify layer-wise distortions using statistical and distributional measures.

The results reveal non-linear breakpoints at low bit-widths, and show that query and key projections are consistently more sensitive than value and output projections.