This is not incremental progress. Over seven days every core bottleneck holding back production LLM deployment got a working, measured improvement. None are theoretical. All run on hardware you have right now.

You can now run 64k context at 73 tokens/sec, run safety guardrails faster than the model itself, get 4x average speculative decode acceptance, run diffusion models in one step at better FID than the original 63 step model, and run any transformers model in vLLM at native speed without porting.

The KV cache bottleneck is broken

For 3 years every KV cache compression method traded memory for quality. All of them applied uniform compression across every token and every layer. That was stupid. Most tokens in your context do not matter. A tiny fraction carry all the information that will ever be retrieved.

DepthWeave-KV does the obvious thing no one did before: it only preserves full resolution for tokens that actually get attended to. It factorizes shared basis across 3-4 neighboring layers, then keeps tiny per-token residuals only where attention probe error crosses a threshold. No retraining required. No calibration. It adapts during generation.

Numbers: 8.3x KV memory reduction. 72.8 tokens per second at 64k context on a single H100. Across every long context benchmark it scores within 0.2% of uncompressed cache. It beats every prior compression method on Needle In A Haystack retrieval accuracy by 19 percentage points at 128k context.

This is not a minor improvement. This removes the KV cache as the limiting factor for long context inference for all existing transformer models.

Speculative decoding stops being a gimmick

Speculative decoding always looked good on paper. In practice average accepted length hovered around 1.7, for a ~30% speedup that was almost always canceled out by overhead. Nobody ran it in production.

That changed this week with the release of Kimi K2.6 DSpark. This is the first speculator that reliably delivers >3.9 average accepted length on real workloads. That translates directly to 2.1x - 2.7x end to end decode speed with zero quality loss.

Note the variance. Acceptance rate tracks task coherence. On structured reasoning tasks you get almost full 7 token acceptance. On general chat it drops. On multiple choice exams it falls apart. That is not a bug. That is how this should work.

The speculator is 3 layers, 6.3GB. It runs on one extra GPU while the base model runs on 8. You add 12% extra hardware for 220% extra throughput. This is the best ROI in inference right now.

Safety guardrails don't need to add latency

All production guardrails today are one of two bad options: fast dumb classifiers that miss 30% of harmful requests, or reasoning based guards that add 200-500ms of latency to every request.

DT-Guard fixes this tradeoff completely. The insight is extremely simple: you train the guardrail with full reasoning traces, but you never run that reasoning at inference. You only train the model to internalize the judgement.

Guardrail Model	Size	Average F1	Inference latency
Llama Guard 3 8B	8B	0.821	112ms
DT-Guard	4B	0.878	47ms
GPT-4o Moderation	Closed	0.882	218ms

DT-Guard beats every existing open source guardrail by 7 F1 points while being 2.4x faster. It even matches closed API moderation quality. There is no tradeoff here. Every deployment should switch to this model immediately.

You can now prune diffusion models and make them better

For as long as diffusion pruning has existed everyone accepted that you lose quality when you remove parameters. This paper proves that is not true.

If you bridge pruning and step distillation correctly you get a model that is smaller, runs in one step, and produces better output than the original full model.

Model	Parameters	Steps	FID ImageNet 512
Original EDM2-XS	124.7M	63	3.53
20% pruned 1-step	98.8M	1	3.12
30% pruned 1-step	88.0M	1	4.26

Read that again. 20% smaller, 63x faster, better FID. This is not supposed to be possible. All prior work said this was a hard tradeoff. It wasn't. Everyone was just doing the order wrong.

vLLM just eliminated model porting

Until this week if you released a new model architecture you had to write a custom port for every inference backend. That is over.

The new transformers backend for vLLM takes unmodified upstream transformers model code, rewrites it at runtime, and runs it at exactly the same speed as hand optimized native vLLM implementations.

This was tested across 4B dense, 32B tensor parallel, and 235B MoE models. In every case it matched or beat the native hand written vLLM implementation throughput.

This is the single most important infrastructure change for LLM deployment in the last 18 months. Model authors will never need to write inference backend code again.

DeepSeek V4 Flash redefines baseline efficiency

DeepSeek dropped V4 Flash this week and it is not getting enough credit for what it actually is: the first model that delivers near frontier performance at a cost structure that makes mass deployment viable.

13B activated parameters. 1M token context. It beats Llama 3.1 70B on every benchmark. It runs on two 80GB GPUs. It will run on one H200 with good quantization.

Unsloth released GGUF quants 12 hours after the model dropped. You can run it right now on llama.cpp at 70 tokens/sec on consumer hardware.

This is the new baseline for every production deployment. There is no good reason to run any 70B class model ever again.

The lost in the middle problem is solved

Long context models still could not reliably retrieve information from the middle of the context window. Everyone knew this. No one had a good fix.

LongCrafter is not an inference optimization. It is a data synthesis pipeline that fixes this at fine tuning time. It builds explicit evidence graphs across long documents, then generates instruction pairs that require retrieving information from arbitrary positions.

Models fine tuned on LongCrafter data have flat retrieval accuracy across the entire context window. There is no longer any penalty for putting information in the middle. This fixes the single most annoying failure mode of long context models.

What this means for production

All of these changes are available today. You can implement every single one of them this month.

Combined they deliver approximately:

• 8x lower memory usage for long context
• 2.5x higher decode throughput
• 0.8x lower latency
• Better safety
• No measurable quality loss

This is not 10% improvement. This is an entire generation shift in inference efficiency. The cost of running an LLM just dropped by 75% in one week.

Most teams have not caught up yet. Over the next 90 days every production deployment will be rebuilt on top of these techniques. The ones that do it first will have an enormous cost advantage.

Closing notes

None of these papers use fancy new architectures. None require new hardware. All of them are just doing obvious things that no one bothered to do properly before.

That is the pattern right now. We are not waiting for new model breakthroughs. We are just finally fixing all the stupid obvious inefficiencies that everyone accepted as unavoidable for the last three years.

There is still a lot of low hanging fruit left.