If you have ever deployed a quantized LLM, you have already seen this.

You run MMLU. You run perplexity. All the numbers look almost identical to the base model. You ship it. Then users start reporting weird bugs. The model will sometimes forget instructions. It will output broken JSON. It will make trivial logical errors that the full precision model never made. No standard benchmark will catch this.

That is not a bug in your implementation. That is the entire field operating on a shared lie. Quantization is not lossless compression. It is not even gradual degradation. It is targeted damage to specific parts of the model, that is invisible to the metrics we currently use.

The illusion of equivalence

Last week a paper from Meta and UC Berkeley formalized exactly this effect. They introduced a metric called correctness agreement: instead of measuring how often the quantized model gets an answer right, measure how often it gives the same answer as the base model, right or wrong.

Across Llama 3, Mistral and Qwen, from 8 bit down to 2 bit:

• Perplexity stays within 1% of base all the way down to 3 bit
• MMLU stays within 2% down to 2.5 bit
• Correctness agreement falls off a cliff at 4 bit. At 3 bit, the models only agree on 62% of answers.

They are not slightly worse. They are different models.

Most importantly this divergence is not uniform. Query and key projection layers are 3-7x more sensitive to quantization error than value or output layers. Quantization damages attention coordination before it damages raw knowledge retrieval. This is why you can ask a Q4 model for the capital of France and it will always get it right, but ask it to follow a 3 step instruction and it will fail half the time.

The 25x speedup that almost worked

This is not abstract research. This is exactly what Anna documented running Llama 3.2 1B on a Jetson Nano.

She did everything right. She built Ollama from source. She enabled GPU acceleration. She tested quant levels. This is the result:

Metric	Q8_0	Q4_K_M
File size	1.5 GB	808 MB
GPU layers loaded	3-9 / 17	17 / 17
Generation speed	1.3 tok/s	30.7 tok/s
971 token test	13m 20s	40s

That is a 25x speedup. It is also completely unusable.

When she tested actual output:

• 2/6 valid correct JSON
• 1/6 valid JSON wrong field name
• 3/6 unparseable garbage

50% failure rate. Perplexity for this quant is within 0.8% of the base model. No standard benchmark would have warned her.

She added retries. That got effective success rate up to ~90%. But as one commenter correctly pointed out: retries only catch broken syntax. They do not catch correct syntax with wrong values. That failure is silent. And it is the most common failure mode of aggressively quantized models.

Capability specific degradation

This pattern holds across every recent independent benchmark. Quantization does not damage all capabilities equally.

The CUHK HPC cluster ran Qwen 3.6 across all quant levels against two benchmarks: GPQA Diamond which tests raw factual knowledge, and Terminal Bench 2 which tests agentic reasoning, tool use and instruction following.

Quant	GPQA Score	Terminal Bench 2 Score
BF16	88.1	72.4
FP8	87.9	71.1
W4A16	87.7	63.8
W4A4	87.2	51.2
Q2	86.3	29.7

Knowledge falls 2% across the entire range. Agentic performance falls 59%.

This is the most important published result about quantization in the last 12 months, and almost no one is talking about it. You can crush a model down to 2 bits and it will still remember almost every fact it ever knew. It will just become completely unable to do anything useful with those facts.

What breaks, and what doesn't

We now have a consistent picture of what survives quantization and what dies:

Capability	Sensitivity to quantization
Factual recall	Very low
General fluency	Low
Instruction following	High
Structured output conformity	Very high
Multi step reasoning	Extreme
Consistency across repeated runs	Extreme

This is not an accident. This directly maps to the layer sensitivity measured in the Berkeley paper. Facts live in the feed forward layers, which are extremely robust to quantization. Reasoning, coordination and consistency live in the attention projections, which are not.

Extreme compression without codebooks

For teams that actually need to go below 3 bit, the standard approaches are running out of headroom. Scalar quantization falls apart. Vector quantization adds codebook lookup overhead that eats most of the memory gain.

This week BiSCo-LLM was published, which skips codebooks entirely. Instead it maps local weight chunks onto a unit hypersphere and stores only the sign bit, plus a tiny residual correction stream. It hits effective 1.6 bit per weight with 92% correctness agreement, which is better than any existing 2 bit quant method.

Notably BiSCo explicitly preserves 8 bits for the query and key projections. The authors did not even try to quantize them below 8 bit. They already knew it does not work. Everyone else will catch up to this in about 6 months.

Pruning MoE models correctly

For MoE models the problem gets worse. Everyone knows you can prune half the experts and almost no benchmark moves. Almost no one knows that all existing pruning heuristics break routing consistency.

MAESTRO, published this week, is the first pruning method that actually models the routing graph instead of just measuring individual expert activation frequency. It retains 10% higher performance at 50% compression than all prior methods, and more importantly it cuts cross task performance variance by 72%.

Pruned MoEs are not just faster. When done wrong they are pathologically inconsistent. They will pass every single benchmark you run on them, and fail catastrophically on one out of every twenty real world requests. No one measures that.

Running a 744B model on 25GB of ram

Last week someone dropped colibri, an engine that runs GLM-5.2 744B MoE on a consumer machine with 25GB ram. It does not do magic. It does exactly what everyone should have been doing for MoE deployment:

• Keep the dense attention layers resident in memory
• Stream experts from disk on demand
• Cache the most frequently routed experts

It gets 0.1 tok/s cold, 0.5 tok/s warm. That is not fast. That is a frontier class model running on hardware that cost $300.

Most importantly the author noted exactly one non negotiable requirement: the multi token prediction head cannot be quantized below 8 bit. At 4 bit acceptance rate drops from 55% to 4%. That is the attention coordination effect again, showing up exactly where everyone said it would.

Stop benchmarking accuracy. Start benchmarking agreement.

Every single result here points to the same conclusion. We have been measuring the wrong thing for 4 years.

Perplexity does not predict production performance. MMLU does not predict production performance. The only metric that correlates with whether a quantized model will actually work in production is agreement with the base model.

If you are evaluating a quantization method, do not measure how often it gets answers right. Measure how often it gives the same answer as the full precision model. Right or wrong. If it disagrees, it does not matter that it is sometimes right. It is a different model.

Closing

Quantization is the most important technology in production LLM right now. It is also the most poorly understood.

We have built an entire ecosystem around the lie that you can take a 70B model, crush it down to 4 bits, and it is still the same model. It is not. It will remember all the same facts. It will sound exactly the same. It will pass every standard benchmark. And it will quietly fail at the exact thing you deployed it to do.

This is not an unsolvable problem. We know which layers are sensitive. We know which capabilities break first. We have new metrics that actually measure the effect. We just have to stop lying to each other about what quantization actually does.

It is not compression. It is behavioral modification. Treat it that way.