If you operate deployed LLM systems, stop what you are doing. Every single major bottleneck in transformer inference got a material, production ready improvement in the last three weeks. None of these are academic toys. All will land in mainline serving stacks before the end of August. You can cut your cost per token by half, or halve end user latency, without retraining models, changing prompts, or switching providers.

We are no longer fighting the same bottlenecks

Up until six months ago there was universal agreement on the order of inference bottlenecks:

1. KV cache memory footprint
2. Attention compute scaling
3. Autoregressive decode serialization
4. Model weight memory bandwidth

That order is now completely inverted. Every one of these constraints was broken this month. None of the old rules for capacity planning apply any more.

Attention is no longer O(N²): AVQ-Attention breaks the old tradeoff

Vector Quantized attention reduced attention complexity from O(N²) to O(MN) last year, but came with an unresolvable flaw. Fixed codebooks allocate identical representational capacity across all key space, regardless of where attention mass actually lands. High importance regions got coarse approximations while empty regions wasted codebook slots.

Adaptive Vector Quantized (AVQ) Attention fixes this. Starting from a small base codebook, it identifies high weight codes during the forward pass and refines only those locations with pre-trained child codewords. Fine grained quantization is applied only where it will actually change output. Coarse quantization remains everywhere else.

The entire refinement runs inside the existing tiled execution pattern of Flash Attention, with 3% fixed overhead. For 128k context windows it delivers 2.1x faster attention at identical perplexity compared to Flash Attention 3. At equal speed, perplexity improves by 12%. A production Triton kernel is already published. vLLM has an open PR to merge it this week.

KV cache compression finally stops being a quality tradeoff

For two years every KV cache compression method required an explicit quality tradeoff. 4-bit quantization and cross layer SVD would throw away 5-15% of model performance at 2x compression. Everyone accepted this as unavoidable.

JoLT changes this. It treats the KV cache as the third order tensor it actually is, rather than a stack of unrelated 2d slices. It applies partial Tucker decomposition only across token and feature axes, then restores discarded residual energy with a Johnson-Lindenstrauss rotated low bit buffer. A single optimizer allocates rank and bit width across all layers under a fixed byte budget.

At 2x compression JoLT produces reconstruction error an order of magnitude lower than all prior methods. GSM8K accuracy, RULER retrieval and perplexity all land within measurement noise of an uncompressed cache. There is no longer any reason to run uncompressed KV cache. None.

Speculative decoding was broken for MoE, until EcoSpec

Everyone thought speculative decoding was a solved problem. It was not, for Mixture of Experts models.

Standard confidence based draft selection causes expert scattering. You pick 8 draft tokens, each routes to a different expert. You end up loading 17 out of 22 experts for verification, completely destroying any speedup you would have gotten from speculation. This effect was almost entirely unreported until this month.

EcoSpec fixes this by adding predicted expert activation cost to the draft selection heuristic. It prefers draft paths that reuse experts already loaded for verification. It does not modify the target model verification rule, and it does not reduce token acceptance rate. On DeepSeek V3.1 671B it delivers 1.62x end to end decode speedup. No other change required. This is the single largest MoE inference improvement ever published.

Diffusion LLMs are no longer just theoretical

The recent dLLM survey finally stops the hype and lays out actual constraints. Parallel generation is real, but you will not see speedups unless you implement diffusion aware caching. Right now raw diffusion LLMs run 15% slower than equivalent autoregressive models. With the optimizations outlined, they will run 2.7-3.2x faster by the end of the year.

Most importantly the paper introduces a standard latency decomposition framework that disentangles the three separate layers of overhead that have been conflated in every prior benchmark:

This framework is the single most useful contribution to inference research in the last two years. Every benchmark published before this paper can be discarded.

Production serving stacks are moving faster than research

Six months ago research papers led production implementation by 6-12 months. That lead has collapsed entirely. Today production stacks implement optimizations before the arxiv paper finishes peer review.

ExLlamaV3 shipped online KV quantization two weeks before the JoLT paper was posted. vLLM merged MoE ticket scheduling the same day the EcoSpec preprint went live. Researchers are now benchmarking against code that shipped before their paper was written.

ExLlamaV3: The quiet reference implementation everyone copies

ExLlamaV3 1.0 landed this month. Almost no one writes about it. Every single other serving stack copies every major kernel optimization from ExLlama 3-6 months later.

This release removes Flash Attention 2 entirely. Its custom attention kernel runs 12% faster on Ampere, 18% faster on Ada. It has zero slowdown for KV quantization. Most users will see 30-40% faster inference just upgrading. This is currently the fastest single GPU inference implementation on the planet, by a wide margin.

vLLM vs Triton: The serving divide is now clear

The two dominant production serving stacks have completely stopped competing. They have diverged into separate use cases and there is no middle ground any more. This is a good thing.

Feature	vLLM	Triton Inference Server	ExLlamaV3
Primary use case	High throughput LLM serving	General purpose multi model inference	Single / multi GPU low latency
MoE support	Excellent	Basic	Excellent
Speculative decoding	Built in	Requires custom backend	Built in
KV compression methods	8	2	6
Open source license	Apache 2.0	Apache 2.0	MIT
Release cadence	Weekly	Quarterly	Biweekly
Supports non LLM models	Partial	Full	No
Community contributors	2100+	700+	120+

If you are only serving LLMs, you use vLLM. If you are running pipelines with multiple model types, you use Triton. That is the entire decision.

Benchmarking is still broken

All of these papers report numbers that are almost impossible to reproduce. No one publishes batch size, load factor, or request arrival patterns. End to end latency depends on every single layer in the decomposition diagram above. Almost all published benchmarks only measure one.

Until we get standard benchmark suites that measure all three axes of overhead, all speedup claims should be treated with extreme skepticism. You will only know what works when you run it on your actual traffic.

What comes next

We are about to cross an important threshold. Very soon, inference cost will no longer be the limiting factor for LLM deployment. The cost of running a 70B model will drop below $0.05 per million tokens before the end of the year.

The bottleneck will move somewhere else. It will move to data, to alignment, to prompt engineering. No one is ready for that.