Nobody talks about this at model launch parties, but 90% of the real work in LLMs happens after the weights are downloaded. If you are running anything at scale right now, you are fighting three bottlenecks, in this order: KV cache memory bandwidth, quantization accuracy loss, and bad cache eviction logic.

This month three papers dropped that directly move the needle on all three. None of them require training a new foundation model. All have hard, reproducible numbers. None are just marketing benchmarks.

The unglamorous bottleneck everyone is fighting

Every LLM serving cluster in the world is currently bottlenecked on memory bandwidth, not compute. You can have all the tensor cores money can buy, they will sit idle waiting for KV cache entries to be loaded from DRAM.

This is not a temporary problem. It gets worse as models get larger. It gets worse as context windows grow. It gets worse as batch sizes increase. Every optimization that reduces memory traffic gives you more effective throughput than any new GPU architecture released in the last three years.

All three papers covered here attack this exact bottleneck.

Fast-TurboQuant: killing multipliers for 1-bit quantization

1-bit quantization is the holy grail for memory bound workloads. It reduces bandwidth requirements by 97% compared to fp16. Until now every practical 1-bit method came with an unacceptable catch.

Original TurboQuant solved the accuracy problem for 1-bit vector quantization, but required a dense random projection step that burned more compute than you saved by quantizing. That projection also required full floating point multipliers, making it completely unusable on edge silicon without hardware FPU support.

Fast-TurboQuant fixes this completely.

How Fast-TurboQuant works

The core insight is that you do not need a true Gaussian projection to condition vectors for quantization. You only need something that satisfies the sub-Gaussian concentration property. A structured fast Johnson-Lindenstrauss transform does this perfectly, and can be executed using only addition and subtraction operations.

There are no tricks here. No hidden overhead. Every operation in this pipeline runs on the integer ALU. On cores without hardware multipliers this runs faster than any other quantization method ever published.

Method	Requires multipliers	Complexity per vector	Speedup vs TurboQuant	Recall@10 delta
Vanilla 1-bit	No	O(d)	1.0x	-12.4%
Original TurboQuant	Yes	O(d²)	1.0x	0.0% baseline
Fast-TurboQuant	No	O(d log d)	19.7x	+1.2%
AWQ 4-bit	Yes	O(d)	0.7x	+2.1%

This is the rare optimization that is both faster and more accurate than the method it replaces.

OpenPangu on Ascend: the first proper NPU quantization study

Virtually every published quantization benchmark runs on NVIDIA hardware. Almost no public data exists for real world performance on Ascend NPUs, despite these chips now running an estimated 35% of all private LLM deployments globally.

This paper ran a controlled, identical test suite across every major post training quantization method on OpenPangu 1B and 7B models running on Ascend 910B1 silicon. No cherry picking. No excluded failure cases.

Quantization accuracy by bit width

The single most important result from this study is one that almost no one talks about: quantization behaviour is not consistent across model sizes.

8-bit weight only quantization is effectively lossless for both models. Average accuracy drop across 18 evaluation tasks was 0.2% for 1B and 0.1% for 7B. There is no good reason to run these models at any higher precision.

4-bit weight only quantization works acceptably for 7B, with 2.7% average accuracy loss. For the 1B model that loss jumps to 11.8% on reasoning, math and code tasks. This is not an implementation bug. Smaller models have less redundant weight space. Compression hits them much harder.

Ultra low bit widths are still not usable. All tested 2-bit methods scored within 3% of random guessing on MMLU. W4A4 SmoothQuant produced infinite perplexity and no usable output at all.

Adaptive KV caching: LRU is bad for LLMs

Every LLM serving stack right now uses LRU cache eviction for KV blocks. Everyone copied this implementation from the first version of vLLM, and no one stopped to verify if it was actually a good policy for this workload.

It is not.

LRU was invented for general purpose workloads in 1966. LLM generation has an extremely predictable access pattern that LRU handles exactly backwards. The first 10% of tokens in any context window are hit constantly for the entire generation run. The most recent tokens are almost never accessed again after the next token is generated. LRU will evict the frequently used old tokens first.

Cache hit rate and latency improvements

The proposed policy does one simple thing: it splits the KV cache into two separate pools. One pool holds frequently accessed blocks. One holds recently accessed blocks. The split ratio is adjusted dynamically at runtime based on observed hit rates for each pool.

That is the entire change. No machine learning. No bloom filters. No fancy data structures.

Improvements are largest on long document QA workloads, where cache thrashing was most severe. Even on real world chat traffic the 2% latency improvement comes for effectively zero engineering cost.

Data points worth noticing

All values are taken directly from the paper evaluations, no extrapolation:

1. Fast-TurboQuant achieves 19.7x sequential speedup over original TurboQuant on x86. On edge RISC-V cores without hardware multipliers this jumps to 72x.
2. 8-bit weight only quantization on OpenPangu 7B on Ascend 910B has <0.3% average accuracy loss across 18 benchmarks.
3. 4-bit weight only quantization on OpenPangu 1B drops reasoning task performance by 11.8%. On the 7B variant this drop is only 2.7%.
4. All tested 2-bit quantization methods for OpenPangu produced valid output, but scored within 3% of random guessing on MMLU.
5. Adaptive recency/frequency KV caching improves hit rate by 10.8% on document QA workloads over stock vLLM LRU.
6. First token latency is reduced 12.6% on the same workloads. For real world chat traffic the improvement drops to 2.0%.
7. W4A4 SmoothQuant on Ascend NPUs produced infinite perplexity during evaluation. No usable output was generated.

Production deployment notes

None of these are free wins. All have tradeoffs you should know before deploying:

• Fast-TurboQuant requires vector dimensions padded to the next power of two. OpenAI embeddings work perfectly. Custom model dimensions will incur minor padding overhead.
• Adaptive KV caching adds approximately 1.2% CPU overhead per request. This is an excellent trade for lower latency, but you will see this on your utilization graphs.
• Do not run 4-bit quantization on any model smaller than 7B. This has been consistent across every study published for 18 months. People keep ignoring this result. Stop ignoring it.
• Fast-TurboQuant reference code currently has an alignment bug on ARM NEON. The patch is posted in the paper issue tracker.

What this changes for your stack

This is what actual progress in LLMs looks like right now. Not 1000B model announcements. Not new benchmark leaderboards. Small, boring, measurable improvements to the infrastructure that runs every single request you serve.

You can deploy all three of these changes this week. All have reference implementations posted with the papers. None require retraining. None require new hardware.

Most teams will not do this. They will keep arguing about which new base model is best. They will keep waiting for the next magic architecture that solves all their problems.

You can beat them all just by fixing the boring stuff.