Three papers landed on arXiv last week that will change how every production LLM team runs MoE models before the end of the year. None got the hype they deserved. All three solve problems that every operator running MoEs has been fighting for 18 months, with no public solution.

If you have ever run out of VRAM halfway through a MoE serving batch, watched an RL fine tune silently collapse after 150 steps, or done the math and realized 75% of your MoE compute is being thrown away on inactive parameters: these papers are for you.

The three problems every MoE operator is fighting right now

Every production MoE deployment currently loses on exactly three failure modes:

1. At serving time, expert weights and KV cache fight for every byte of VRAM. You either cap batch size to unusable levels, or quantize all weights and destroy output quality. There was no middle ground.
2. RL fine tuning for MoEs runs at half theoretical maximum throughput. Everyone accepted you could not run rollouts faster than FP8 without training collapsing. No one published why this happened.
3. All existing MoE scaling laws waste ~70% of training compute on parameters that are never activated for any given token. Everyone knew this was stupid. No one had a working alternative.

All three problems got working, production ready solutions published within 72 hours of each other.

PagedWeight: stop quantizing all experts the same way

PagedWeight starts with an observation so obvious it is embarrassing no one built this already. Static quantization applies the same bit width to every expert in the model. This is catastrophically wasteful.

In real world serving traffic, 92% of all requests hit 11% of the available experts. The remaining 89% of experts sit idle 99% of the time. There is no reason to keep cold experts loaded at full precision.

PagedWeight implements runtime precision paging. It maintains a heat score for every expert, and dynamically adjusts quantization level based on current VRAM pressure. When KV cache usage grows, cold experts are progressively demoted first to INT8, then INT4. When load drops, they are reloaded at FP16 before the next request arrives.

No quantization is applied to hot experts. No user ever sees reduced quality.

PagedWeight benchmark results

All tests were run on Mixtral 8x7B with 1024 token context, real production request traces.

At 28% of the original memory budget, PagedWeight maintains statistically identical accuracy to full FP16. Static quantization falls off a cliff below 50% memory, as popular hot experts get quantized and introduce visible errors.

The 1.94x throughput improvement is not a synthetic number. This is real user traffic. This will cut serving cost per MoE token almost in half.

The unspoken RL rollout collapse problem

Every team that tried to run RL fine tuning on MoEs ran into this bug. No one talked about it publicly.

If you run rollout generation at any precision lower than FP8, training will quietly die after approximately 150 steps. Loss stops moving. Pass rate flatlines. No error, no warning, no gradient explosion. The model just stops learning.

Every production team accepted this as an unwritten law of nature. Everyone ran rollouts at FP8, and ate the 40% throughput penalty.

This week the QUADS paper finally documented exactly why this breaks. And fixed it.

Why NVFP4 was killing your MoE RL training

Everyone blamed weight quantization error. They were wrong.

Weights are quantized once on the trainer, then copied exactly to every rollout worker. Quantization error for weights is identical on both sides of the training loop. The model sees exactly the same error during rollout and gradient update. It learns to compensate perfectly.

Activations are not copied. They are computed independently on every worker.

NVFP4 uses a 1 exponent 1 mantissa format. Quantization error here is not random noise. It is biased. That bias drifts by tiny amounts between rollout workers and the trainer. Over training steps that gap exponentiates. After ~150 steps the log probability gap between rollout and trainer is large enough that all policy gradient signal vanishes completely.

That was the entire bug. No one checked this for 18 months.

QUADS implementation and results

QUADS applies two extremely simple changes to fix the drift:

1. On the trainer side, run fake quantization only for weights. Leave activations at full precision. This aligns the expected error distribution.
2. On the rollout side, apply a tiny per-channel residual offset to activations that cancels 97% of the steady state bias drift.

All native W4A4 GEMM operations remain completely untouched. You keep 100% of the hardware acceleration.

Method	Rollout precision	Average pass@1	Relative throughput	Training collapses at step
BF16 baseline	BF16	52.1	1.00	Never
Naive FP8	FP8	50.7	1.27	Never
Naive NVFP4	NVFP4	12.2	1.47	147
QUADS NVFP4	NVFP4	51.8	1.43	Never

QUADS delivers 16% higher rollout throughput than FP8, with identical stability and accuracy to BF16. This will cut RL fine tuning cost for MoEs by one third overnight.

Loopie: looped MoEs finally work

For five years everyone knew that looping the same transformer block N times should be as good as N unique blocks. It never was. Every controlled test showed that given equal compute budget, you always got better results just making the model larger.

Loopie breaks that scaling law.

The 20B total parameter / 2B active Loopie model beats a vanilla 30B MoE trained on exactly the same number of FLOPs. It achieved gold medal performance on the 2025 IMO and IPhO benchmarks without tool use.

This is not a 2% incremental gain. This invalidates the core scaling assumption every LLM team has operated under since GPT-3.

The only change that made looped MoEs work

Almost nothing was changed from standard MoE architecture. The entire improvement comes from two additions:

They added one linear exit gate per token, and a 128 byte per-token state buffer that is passed between loop iterations. That is it. No modified attention. No new activation functions. One gate and 128 bytes of state. That was the missing piece for five years.

Tokens exit the loop as soon as they have completed computation. Easy tokens run 2 iterations. Hard tokens run up to 32. Average active parameters remain fixed at 2B regardless of loop depth.

What this means for production next quarter

None of this work is theoretical. All three papers have reference implementations posted. None require custom hardware. All run unmodified on existing A100 and H100 cards.

By the end of Q3 2026 every major LLM provider will be running one or more of these techniques.

Serving cost per MoE token will drop by roughly half. RL fine tuning run times will drop by 40%. And we will start seeing 2B active parameter models that perform on par with current 30B models.

This is not the next big flashy model launch. This is the boring infrastructure work that was holding everything else back. Everyone was waiting for these fixes. They all arrived the same week.

Open remaining gaps

There are still untested edges. PagedWeight has not been validated with continuous batching above 256 concurrent requests. QUADS has only been run on code and math benchmarks, not conversational alignment. Loopie has not been scaled above 20B total parameters.

None of these are fundamental blockers. These are just the things someone will post working fixes for next month.

If you are running MoEs today, go read these papers this week. You will be running them in production before the end of the month.