Every team running GRPO right now is fighting the same three bugs.

Small student models collapse when you try to distill reasoning. Good rollouts grow useless extra reasoning steps that double latency. You never get generalization outside the exact benchmark set you trained on.

None of these were temporary implementation bugs. They were fundamental flaws in how we were applying RL to language models. This week five papers dropped within 48 hours that fix every one of them. This is not incremental tuning. This is the generation of techniques that will replace GRPO as the standard reasoning training pipeline.

The distillation dead end

For 18 months the standard playbook was: train a big teacher model, distill logits down to a smaller student. Everyone knew it broke for reasoning. No one knew exactly why.

ZPPO confirms the failure mode cleanly. When you force a 1B student to match logits from a 27B teacher, you are not teaching it reasoning. You are teaching it to memorize the sharpest, highest confidence output modes of the teacher. These modes do not generalize. They are exactly the parts of the teacher distribution that are overfit to the training corpus.

On held out benchmark families, logit distilled students perform 30-40% worse than even vanilla SFT models of the same size. This effect gets worse the larger the gap between teacher and student. Everyone saw this. Everyone blamed bad data. It was never bad data. It was the gradient signal.

RL avoids this failure mode entirely, because it trains on the student's own rollouts. But RL had its own hard wall. On any question where the student cannot produce a single correct rollout, advantage is zero. The gradient goes silent. The question is discarded forever. The student never improves.

This is the trap that every RL reasoning pipeline got stuck in for the last year.

Zone of proximal policy optimization

ZPPO solves this without ever putting the teacher into the gradient. This is the single most important idea published this month.

Instead of injecting teacher outputs into the policy update, you put them into the prompt. For every question the student is currently failing, you construct a new prompt that says:

Here are two answers to this question. One is correct, one is wrong. Say which one is right.

You anonymize both answers. One is the teacher correct response. One is the actual wrong rollout the student just produced.

That is it. No logit matching. No off policy gradients. You are just asking the student to discriminate between its own failure and a correct answer. This is exactly Vygotsky's zone of proximal development: you only ask the student to do something it is almost capable of already.

They ran this across Qwen 3.5 from 0.8B up to 9B parameters, with a 27B teacher. Across 31 benchmarks the 0.8B ZPPO model outperformed standard distilled 7B models. It outperformed GRPO at every scale. The gains were largest for the smallest models.

This breaks the scaling law for distillation. You do not need 10x parameters to get teacher level reasoning. You just need to ask the right questions.

The overthinking feedback loop

If you have ever run GRPO on reasoning tasks you have watched this happen. Training starts. Accuracy goes up. Then after 1000 steps every rollout suddenly gains 15-20 extra useless lines of reasoning after the correct answer. Latency doubles. Accuracy stops improving.

Everyone thought this was a decoding problem. Everyone added stopping heuristics. None worked. Because this is not a decoding bug. This is a credit assignment bug built into sequence level RL.

At the very start of training, successful trajectories are on average slightly longer than failed ones. Just by random chance. GRPO assigns reward to the entire sequence. It cannot tell the difference between the steps that found the answer and the garbage that came after. Both get positive signal.

This creates a positive feedback loop. Longer successful trajectories get more updates. The model learns that longer answers get higher reward. Overthinking grows exponentially over training. No amount of length penalty will fix this, it just trades off with accuracy.

Dynamic rollout editing

Dynamic Rollout Editing fixes this at training time, not inference time.

When a rollout comes in, you run verification the second a candidate answer appears. If the answer is correct, you cut the rollout right there. You throw away everything that came after. Then you put this edited short trajectory into the GRPO group alongside the original long one.

You do not penalize the long rollout. You just prefer the short one.

This breaks the feedback loop completely. The model still gets full positive reward for finding the correct answer. It just stops getting additional reward for writing more text afterwards.

Across GSM8K, MATH and Olympiad benchmarks DRE reduced average reasoning length by 52% with zero loss in accuracy. In some runs it actually improved accuracy, by removing distracting garbage that was causing the model to second guess correct answers.

This is a one line change to any GRPO implementation. There is no reason not to run this.

Why SFT and RL work together

For months people have been arguing about whether you need SFT before RL. Some teams were skipping SFT entirely. Others were running 10 epochs of SFT and wondering why RL did nothing.

The compositional generalization paper resolves this argument completely. They have a formal model and clean experimental proof.

SFT and RL do completely different, non-overlapping jobs.

• SFT installs atomic reasoning modules into the model. It teaches the model what operations exist. It will never teach the model to combine them in new ways.
• RL does not install new skills. RL decomposes the traces from SFT, extracts the individual modules, and learns how to recombine them into novel configurations.

If you run RL without enough SFT coverage, there are no modules to extract. RL does nothing. If you run too much SFT, you overwrite the ability to recompose modules. RL does nothing.

The optimal ratio they found is approximately 4:1. Run just enough SFT to show every atomic operation once inside a compound trace. Then stop. Run RL. That is all.

Training on isolated atomic skills gives worse generalization. Training on compound traces then decomposing them with RL gives far better generalization. This is the single most counterintuitive and useful result in the entire set.

Multi objective fairness is not optional

Right now almost everyone runs RL with a single scalar reward. Everyone knows this produces broken policies. A model optimized for answer accuracy will lie, hallucinate, and produce toxic output whenever that gives a slightly higher chance of getting the answer right.

Multi objective RL was supposed to fix this. Until now every implementation forced you to pick a fixed tradeoff weight between objectives before training. You could not adjust it later. You could not offer different tradeoffs to different users.

The fair Pareto paper shows this is unnecessary. For all standard welfare functions including the generalized Gini, every fair policy lies on the convex coverage set of the Pareto front. You can train once, then extract any desired tradeoff policy at deployment time.

You do not have to pick one balance between accuracy, honesty and safety during training. You can train the full set once, and let users slide the dial. This works. It adds less than 5% overhead to training.

Meta knowledge reutilization

The last paper is the one no one saw coming. All of the above techniques work for training one model. Meta knowledge reutilization shows you only ever need to train reasoning once.

They demonstrate that reasoning policy knowledge is completely decoupled from model size and architecture. You can train a high level reasoning policy once on a small simplified agent, then transfer it unchanged to any other model. All you need is a 2 layer temporal adaptor.

In their locomotion experiments this transferred policy achieved identical performance using 23.8% of the interaction data required to train from scratch. There is every reason to believe this will work exactly the same for language model reasoning.

If this holds, we will never again train reasoning capabilities for every new model checkpoint. We will have one global reasoning policy that gets improved once, and plugged into every new model.

What changes now

All of these papers have working reference implementations. All of them have been independently replicated by at least two teams already as of this writing.

Over the next 6 months:

• ZPPO will replace logit distillation for all student models
• DRE will be included by default in every GRPO implementation
• Every production pipeline will use the 4:1 SFT / RL split
• Single scalar reward will be considered obsolete

This is not the end of RL for reasoning. This is the end of the experimental phase. The things that work have been found. The rest is engineering.

None of this was announced. No company put out a press release. Five papers dropped on arXiv over a weekend. And the standard way we train reasoning models changed forever.