The takeaway

All five works released this week solve real, painful problems that people are actually dealing with right now running large models. No toy benchmarks. No 1% improvements that vanish out of distribution.

If you train models: stop what you are doing and test DMuon today. If you run inference: deploy JetSpec next sprint. If you build recurrent architectures: CARVE is the new baseline. If you ship image models: TMP will let you run 80B diffusion on consumer GPUs.

That is not hype. That is what the numbers say.

The broken state of recurrent gating

For two years every recurrent linear attention architecture used the exact same gating pattern. Every single one erased memory before looking at what was already stored.

Nobody noticed this was broken. Everyone just copied the gate implementation from GDN-1, then GDN-2, then every derivative that followed.

The gate only saw the incoming token. It had zero information about the memory state it was about to modify. This was not an oversight. Everyone assumed this was required for the fast chunk parallel training solver.

It was not required. It was just a mistake.

CARVE fixes this with one change: erase only on the key axis. That is the entire trick.

This one change fixes three separate defects that nobody had even properly identified:

1. Memory blind gating is removed. The gate now sees the existing recurrent state, which is already sitting in GPU memory anyway. No extra compute. No extra parameters.
2. The entire value projection gate is removed. 19% of all parameters in every GDN-2 head were completely wasted.
3. Most importantly: this change is mathematically compatible with the WY triangular solver that makes recurrent training fast. All prior attempts to fix gating broke training parallelism.

CARVE benchmark results

This is how you present results. No asterisks. No fine print. Same initialization. Same training run. Same number of tokens. Same hardware.

Model	Parameters	WikiText Perplexity	Peak Memory	Throughput
GDN-2 1.3B	1.3B	15.90	100%	100%
CARVE 1.3B	1.05B	15.72	87%	99.6%

That is better perplexity, 13% lower memory, 19% fewer parameters, and effectively identical throughput. This is not a tradeoff. This is a strict improvement across every axis.

The authors also ran every RULER retrieval probe. CARVE beat every existing recurrent model on every single one. There is no reason left to use GDN-2 for anything.

Muon was too slow. That is fixed now.

Muon was the most interesting optimizer released in the last three years. It consistently beat AdamW on every workload people tested. Almost nobody used it.

Vanilla Muon took more time to run the optimizer step than the entire forward and backward pass combined. It added 120% overhead to every training step. Everyone looked at the numbers, said that is very nice, and went back to AdamW.

Two separate teams fixed this in the same week.

First DMuon. This is not a new algorithm. This is just someone finally implementing Muon correctly for distributed training. No approximations. No changes to the update rule. Just good engineering, proper kernel fusion, and correct batching across layers.

DMuon runs the optimizer step between 6.85x and 163x faster than the reference implementation. End to end step overhead over AdamW is now between 2% and 11%. That is noise. You will not notice it.

This is a drop in replacement. You change one line in your training config. That is it.

Hierarchical Muon trades global coupling for speed

HiMuon is the actual algorithmic improvement.

Full Muon runs Newton-Schulz over the entire weight matrix. This couples every single weight value to every other one in the matrix. For large matrices this is extremely expensive.

HiMuon splits the matrix into tiles. It runs exactly the same Newton-Schulz operation independently inside each tile. Spectral interactions are preserved inside tiles, discarded across tile boundaries.

For tile sizes of 8 or 16, training behaviour is indistinguishable from full Muon on every tested workload. Step time drops to within 10% of AdamW.

This is not an approximation. This is a different optimizer. It just happens to behave almost exactly like the full version, while running 2-3x faster.

Tree pruning for diffusion models

Everyone has been pruning transformers for years. Nobody had got it working properly for diffusion transformers and MoE image models until this week.

TMP works. That is the news.

They pruned HunyuanImage 3.0 75% from 80B down to 20B parameters. Human preference scores dropped 1.8%. That is within measurement error for most evaluation runs.

The pruned 20B model runs on a single 24GB RTX 4090.

Let that sink in. Six months ago you needed 8 A100s to run this model. Now you run it on consumer hardware you can buy for $1800.

They also pruned Z-Image Turbo 33% from 6B to 4B. No measurable quality loss at all.

This method works on already distilled models. You run this as the very last step before shipping. There is almost no downside.

JetSpec breaks the speculative decoding ceiling

Speculative decoding hit a hard wall at ~3x end to end speedup. Everyone accepted that this was the theoretical limit.

It was not the limit. It was just a bad drafting algorithm.

JetSpec drafts an entire valid causal tree of 32 candidate tokens in one single forward pass. No sequential steps. No inconsistent branches.

Method	MATH-500 speedup	Chat speedup	Max TPS B200
Baseline	1.0x	1.0x	112
Standard SD	2.72x	1.91x	301
DFlash	3.81x	2.77x	428
JetSpec	9.64x	4.58x	1017

This is lossless. The output is bit identical to running the full model normally. There is no quality tradeoff.

This is the biggest inference improvement released in the last two years. Code is available right now. It works with every existing LLM.

What none of the papers say

None of these works are incremental improvements. Every single one resets the baseline for an entire subfield.

Nobody will be running vanilla Muon twelve months from now. Nobody will be using GDN-2. Nobody will ship an unpruned 80B image model. Nobody will run speculative decoding without parallel tree drafting.

All of this happened in one week.

The thing that nobody is pointing out is that every single one of these improvements came from removing things. CARVE removed parameters. HiMuon removed global coupling. DMuon removed bad implementation overhead. TMP removes 3/4 of an image model. JetSpec removes sequential drafting steps.

Almost all progress in efficient ML right now is not adding clever new things. It is finding all the useless garbage that everyone copied from each other for three years, and deleting it.

Outstanding open questions

There are still untested edges for every method covered here.

HiMuon has only been tested up to 70B parameters. We do not know if tile boundary effects will appear at 400B+ scale.

CARVE has only been run at 1.3B parameters. No one has yet scaled it to the 7B+ range where recurrent architectures start competing directly with standard transformers.

TMP pruning ratios are still manually tuned. No one has published an automatic schedule that picks the optimal pruning ratio per layer.

JetSpec speedup falls off for very short prompts. The team has not yet published what causes this effect.

All of these are solvable problems. None are fundamental limitations.

References

1. CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention http://arxiv.org/abs/2606.27229v1
2. Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization http://arxiv.org/abs/2606.27216v1
3. DMuon: Efficient Distributed Muon Training with Near-Adam Overhead http://arxiv.org/abs/2606.27153v1
4. TMP: Tree-structured Mixed-policy Pruning for Large-scale Image Generation and Editing http://arxiv.org/abs/2606.27089v1
5. JetSpec: Speculative Decoding with Parallel Tree Drafting https://github.com/hao-ai-lab/JetSpec