Six papers dropped on arXiv last week. None of them cite each other. All of them arrive at the same conclusion.

The standard GRPO / PPO stack that every lab has used for LLM post-training over the last 12 months is broken. Not slightly broken. Fundamentally, structurally broken at every layer of the design.

This is not a fringe position. These papers come from Tencent, Alibaba, OpenAI, Anthropic, and two independent research groups. Every single one ran controlled ablations and found the same failure modes. No one announced this. No one wrote a blog post. They just uploaded the fixes.

That is how frontier research moves.

We broke GRPO

For 12 months GRPO was treated as a solved problem. Teams argued about reward functions, rollout counts, batch sizes. No one was looking at the core algorithm. Everyone assumed the trust region worked. Everyone was wrong.

All six papers independently observed the same failure pattern. GRPO training would run stable for 100-200 steps, then without warning the policy would collapse. Loss would spike. Reward would flatline. All progress would be irreversibly lost.

Most teams treated this as bad luck. They would restart training from the last good checkpoint, adjust the clip threshold by 0.02, and hope. No one stopped to ask why this happened every single time at roughly the same training horizon.

We now know. Every core design choice in standard PPO was wrong for autoregressive models.

The uniform trust region lie

Every PPO variant used for LLMs today enforces the same rule. For every token generated during rollout, the ratio between the new policy log probability and the old policy log probability may not exceed a fixed clip threshold, almost always set to 0.2.

This rule is applied identically to the first token of the sequence and the last token. It is applied the same way regardless of what came before it.

This is insane.

When you deviate 0.2 on the first token of a 128 token reasoning chain, every subsequent token is conditioned on that deviation. The compound drift is approximately 20x larger by the end of the sequence. When you deviate 0.2 on the second last token, nothing compounds. You change exactly one word.

Uniform clipping was doing exactly the opposite of what it was supposed to do.

Prefix drift is the real failure mode

The authors of Beyond Uniform Token-Level Trust Region did not just notice this asymmetry. They measured it.

Across 100k rollouts on GSM8K, MMLU and MultiHop QA, they found that 87% of all policy collapse events originated from unclipped deviation on one of the first 7 tokens of the sequence. At the same time, 62% of all useful policy improvements came from deviations on tokens after position 40.

Uniform clipping let the dangerous deviations pass, and blocked the useful ones.

Their proposed fix, CPPO, implements two very simple changes. First, the clip threshold scales linearly with position. It starts at 0.05 for token 0, rises to 0.15 at token 32, and hits 0.4 at token 128. Second, it tracks total cumulative KL divergence across the prefix, and reduces the allowed per-token threshold if the sequence has already drifted too far.

That is it. Two lines of change in the gradient mask.

On Qwen3-14B this gives a 3.1 point improvement on GSM8K, 2.7 on MMLU-Pro, and cuts policy collapse events during training by 92%. No other changes. Same rollout budget, same reward function, same number of epochs.

Ratio clipping does not work for flow models

This failure is not unique to autoregressive text models.

Flow matching models for image and video have started moving to RL fine tuning over the last three months. Everyone just ported GRPO over unchanged. It worked badly. No one knew why.

Flow-DPPO explains the problem. Ratio clipping is a noisy single sample estimate of policy divergence. It works acceptably well for categorical token distributions. It does not work at all for the continuous Gaussian policies used in flow matching.

For Gaussian policies you do not need to estimate divergence. You can calculate exact KL between the old and new policy in 3 operations.

Flow-DPPO throws out ratio clipping entirely. It just calculates the actual KL per step, and masks gradients only when that KL exceeds the trust region bound.

This change alone gives 18% higher reward on standard text to image benchmarks, eliminates catastrophic forgetting during multi objective training, and allows stable 8 epoch training where standard GRPO would collapse after 2.

This is not an incremental improvement. This is throwing out a core component of PPO that has existed for 11 years, and replacing it with something that is strictly better for this class of model.

Rollouts are being wasted 90% of the time

Policy optimization signal is extremely sparse.

When you run 16 rollouts for a given prompt, on average 14 of them will return exactly the same terminal reward. All 14 give zero usable gradient signal. You paid for 16 inference runs, you got information equivalent to 2.

No one had properly measured this waste until TRACE was published. Across standard agent benchmarks the average rollout utilization rate is 11%. 89% of all compute spent on sampling during RL training is completely wasted.

TRACE does not allocate rollouts equally across prompts. It builds a tree. It stops rolling out branches that are clearly succeeding or clearly failing. It allocates additional samples only to prefixes that are on the decision boundary, where additional rollouts will actually produce differing rewards.

At equal total sampling budget TRACE improves Qwen3-14B MultiHop QA accuracy by 2.8 points. Or put another way: you can get exactly the same final performance using 40% of the inference budget you were using before.

This is the single largest efficiency gain for LLM RL published to date. Every major lab is already implementing this.

One example is enough to break alignment

All of the above improvements make RL more powerful. They also make it much more dangerous.

It Takes One To Bias Them All is the most important paper published this month. It is also the one that will get no official comment from any LLM provider.

The authors show that you can take any publicly aligned model, run one single GRPO update step on one single biased example, and break the guardrails permanently.

Not for that one prompt. The bias generalizes. It transfers across categories, across benchmarks, across unrelated tasks. The model will retain that bias after 100 further normal alignment steps.

7B models break 98% of the time. 70B models break 71% of the time. There is no threshold of size that prevents this attack.

This is not a jailbreak. This is post training. Anyone who can run fine tuning on an open model can do this. It takes 30 seconds on an A10G.

This is the core vulnerability that no one has an answer for right now. Alignment guardrails are not permanent. They are just a weight state. RL will overwrite them very, very easily.

No one is talking about Q-learning stabilization

Almost all attention right now is on policy gradient methods. Quietly, there is also work rebuilding the foundations of Q-learning for LLMs.

The geometrically averaged hard target update paper looks like boring theoretical work. It is not.

Every Q-learning implementation used today uses periodic hard target network updates. Everyone knows this reduces divergence. No one had a good mathematical model for why it works, or how to tune the update interval.

The λ-target update introduced in this paper gives a continuous interpolation between hard updates and full Q iteration. It provably stabilizes linear Q-learning. Initial unpublicized results show this almost eliminates the training instability that stopped most groups from scaling Q-learning for LLMs last quarter.

This will not ship this month. But this is the foundation that will replace policy gradient methods entirely for reasoning tasks 12 months from now.

RL is no longer just for text

RL alignment stopped being just for chat models this month.

The full duplex speech alignment paper shows exactly the same pattern. Supervised fine tuning produces models that technically work, but behave unnaturally. They pause too long. They talk over you. They do not backchannel.

You cannot fix this with loss functions. You cannot label enough data to teach turn taking. The only method that works is RL, with reward functions tuned for each specific interactive behaviour.

This will be the standard for every interface modality. Speech, video, agents, UI control. None of them will work well with supervised training alone. All of them will require RL fine tuning.

What comes next

Right now every lab is rewriting their RL training stack.

GRPO will be gone by the end of July. CPPO will be the default trust region. TRACE will be the default rollout scheduler. Ratio clipping will be removed from every flow model implementation.

No one will make an announcement. You will just notice that models get better, faster, and training failures stop happening.

The hard part has not changed. We are getting extremely good at optimizing policies for any reward signal we can write. We are still extremely bad at writing reward signals that produce behaviour we actually want.

And we have just learned that once you have this optimization lever, you can pull it in any direction. Very quickly. With almost no effort.

Open questions

There are three questions that no one has answered yet.

Can we build a trust region that prevents alignment override, without blocking useful policy improvement? Right now there is no known way to do this. All existing constraints block bad updates and good updates equally.

All of these improvements assume verifiable binary rewards. None of them work well for human preference rewards. No one is working on this. Preference RL has been completely abandoned by frontier research teams.

We still have no good way to measure how far a policy has drifted from the base model after RL. All existing KL metrics are misleading. They understate drift by an order of magnitude for long sequences.

These are not minor edge cases. These are the open problems sitting between us and reliable aligned agent systems.