Every production LLM deployment running today wastes 75% of its GPU hardware.

For 18 months the entire field was busy building larger models. No one fixed the actual bottleneck. For any model over 7B running batch size greater than 4, generation latency is dominated by KV cache memory and bandwidth, not FLOPs. On NVIDIA B200 GPUs, 90% of available HBM bandwidth is consumed moving KV entries during token generation. Actual FLOP utilization sits at 17% for most production stacks.

Three papers landed on arXiv within 48 hours of each other this week. All three arrived at the exact same core insight from completely different directions. An independent benchmark dropped the same day confirming the numbers. Combined they obsolete almost all LLM serving work published before June 2026.

The serving bottleneck no one advertised

You will not see this mentioned in model release blog posts. A 1M context window is useless if you can only serve one user per A100.

Before this week, standard PagedAttention running on vLLM 0.7 could run 3 concurrent 128k context Llama 3.1 70B users per 80GB A100. It could run exactly one 1M context user. This was not a temporary implementation flaw. This was a fundamental limit of the abstraction everyone had agreed to use.

For three years every serving system treated the KV cache as an opaque, homogeneous blob. Every token, every attention head got exactly the same memory allocation, same retention policy, same precision. No one checked if this made sense.

We stopped treating all attention heads the same

This is the unifying discovery across all three papers. This is the thing that changes everything.

RedKnot ran profiling across 12 production models from 7B to 229B parameters. They found:

• 31% of attention heads never attend to tokens older than 128 positions
• 22% of heads only ever attend to static prefix tokens
• 36% of heads exhibit predictable periodic attention patterns
• Only 11% of heads ever use more than 20% of the declared context window

The remaining 89% of heads were just wasting HBM. They were allocated full context cache they would never read. No one had ever measured this properly at scale. Everyone just assumed all heads needed full context.

Every optimization that follows flows directly from this observation.

Tangram: non-uniform KV cache finally works

Researchers have known for two years you could drop low value KV entries and preserve almost all accuracy. Every single prior attempt to deploy this failed.

Non-uniform KV compression destroyed request scheduling. It caused catastrophic memory fragmentation. Kernel utilization dropped by 40%. You would end up with lower net throughput than just running full cache. Everyone quietly gave up on the approach.

Tangram fixes the systemic problems, not just the compression problem. It uses three simple mechanisms:

1. Deterministic budget allocation assigns a fixed static memory footprint to each head once at model load time. There is zero runtime allocation overhead, zero prefill stalls.
2. Head group page clusters heads with similar retention demands and manages them with independent vectorized page tables. Physical memory reclamation hits 97%.
3. Ahead of time load balancing uses static head profiles to distribute work across SMs before any requests arrive. No runtime balancing overhead.

Experimental results show 2.6x throughput over standard vLLM, with less than 0.1% delta on all standard accuracy benchmarks. It works on every existing transformer model. No fine tuning required. No changes to model weights.

The code is public right now. You can drop this into your serving stack this week.

RedKnot: breaking the monolithic KV abstraction

RedKnot goes much further. It throws out the entire KV cache abstraction that the industry has used since GPT-2.

Instead of one monolithic cache shared across all heads, RedKnot decomposes the cache entirely per head. Each head gets its own retention policy, its own precision, its own eviction logic, its own memory placement.

This is not just another optimization. This is a new base abstraction that solves every active KV research problem at once. Prefix reuse, hot/cold separation, distributed cache placement, position independent reuse. All work cleanly with this one change, no per-case hacks required.

On 1M context Llama 3.1 70B, RedKnot runs 3.1x higher concurrency than standard PagedAttention. Log probability delta between outputs is 0.003. No human or evaluation suite can detect the difference.

Vortex: sparse attention stops being a research toy

Sparse attention had exactly the same failure mode. Everyone knew theoretical throughput gains of 3-5x were possible. No one could deploy it.

Every existing sparse attention implementation was hand tuned for one model, one context length, one GPU generation. Iterating on a new pattern required 2-3 weeks of CUDA engineering. Researchers stopped experimenting.

Vortex fixes this. It exposes a simple Python embedded frontend for describing arbitrary sparse attention patterns. It compiles these definitions down to kernels that perform within 5% of hand written optimized CUDA. You can prototype and deploy a new sparse attention algorithm in an afternoon.

The result that almost everyone skipped over: the authors let a standard LLM agent generate and test sparse attention patterns. The best pattern the agent found beat every human designed algorithm published to date. It delivered 3.46x higher throughput than Flash Attention 3 with no measurable accuracy loss.

On the MLA based GLM-4.7-Flash they hit 4.7x throughput. On the 229B parameter MiniMax-M2.7 running on B200 they hit 1.37x throughput. That is a 37% gain on a state of the art 200B+ model. That result alone would have been the biggest serving paper of the year.

Quantization benchmarks: what actually works for KV cache

The same day all three papers landed, an independent developer posted the first proper unbiased benchmark of KV cache quantization. He ran 75 test pairs across Qwen 3.6 27B up to 512k context.

This benchmark is more reliable than every published academic paper on this topic. There are no hidden cherry picked test cases. There is no marketing agenda.

The final results are unambiguous:

• Q8 KV: zero measurable accuracy loss, 1.9x memory reduction
• Q6 KV: 0.1% MMLU drop, 2.6x reduction
• Q5 KV: 0.8% drop, 3.1x reduction
• Q4 KV: 3.7% drop, 3.9x reduction
• KVarN: beats all uniform quantization, 2.9x reduction with 0.2% accuracy drop
• TCQ and TurboQuant perform worse than plain Q6 at every context length.

Stop using Q4 KV for anything user facing. Stop using any fancy branded quantization method. Run Q6 or KVarN. That is the entire takeaway from 1000 hours of benchmarking.

What you should deploy this month

These are not theoretical research results. These are production ready changes you can apply right now.

If you are running vLLM or Text Generation Inference today:

1. Immediately switch KV cache from FP16 to Q6. You get 2.6x more concurrent sessions for free. No user will notice.
2. Patch Tangram head budgeting. This will give you another 2.1x on top. Total ~5.5x throughput on the same hardware.
3. Do not deploy sparse attention yet. Wait 6 weeks for Vortex to stabilize. It is good but still has sharp edges around variable length requests.
4. Ignore every other KV optimization paper published before June 2026. All of them are obsolete.

On an 80GB A100 you will go from 3 concurrent 128k Llama 70B users to 17. That is not a minor improvement. That is a generational step change.

The end of the context window arms race

This is the real shift that no one has commented on yet.

For the last two years every model release competed on maximum advertised context window. No one could actually serve that window at scale. It was pure marketing.

With these changes you can run 12 concurrent 1M context users per B200. You can run 4 per A100. That is production usable.

Model vendors will stop competing on maximum context length. They will start competing on how efficiently their attention heads use cache. We just moved the goalposts.

Within 12 months every good foundation model will ship with a per head cache budget profile alongside the weights. Models will be explicitly trained for good cache behaviour.

What breaks next

None of this is free. We have just ripped out the foundation layer of LLM serving infrastructure. Everything built on top will now have to change.

Scheduling will stop being done at the request level. It will be done at the attention head level. Load balancing will operate on sub request granularity. Billing models will stop being priced per token.

We will very quickly see serving systems that dynamically recompile attention kernels per user session. We will see systems that adapt cache policy based on the actual observed behaviour of the user's conversation.

No one has tested this combination with MoE models yet. All published results are on dense transformers. No one has measured performance beyond 8 GPU clusters. No one has run the full stack of Tangram + RedKnot + Vortex together. Early unofficial tests suggest total throughput gain will land between 7x and 9x.

Closing note

All of this landed in the same 48 hour window. Almost no one outside the small group of people that run LLM infrastructure has noticed.

There will be no big announcement. No fancy product launch. No press release. Next quarter every major LLM provider will quietly roll these changes out. You will get lower latency, longer context, and they will pay half as much for GPUs. No one will tell you why.

This is how progress actually works in this field. The big flashy announcements are almost always marketing. The real changes land unannounced on arXiv on a random Tuesday.

If you operate LLM infrastructure this is the most important week of the last two years. Go read the papers. Run the benchmarks. Update your stacks. Everyone else will catch up in six months.