We are no longer in the GRPO era

On June 19 2026, nine related papers landed on arXiv within 11 hours. None of them got the normal twitter hype cycle. Every single production LLM team is currently porting at least one of these techniques this week.

Six months ago, everyone was just implementing GRPO and arguing about clip values. That phase is over. All the open problems people were complaining about on internal slack channels got solved. Most people just haven't noticed yet.

None of these papers are minor 0.3% benchmark gains. Every one of them removes a hard limit that everyone had accepted as fundamental. This was the week LLM RL stopped being research and became production engineering.

Entropy collapse was a bug, not a fundamental tradeoff

Everyone running RL on LLMs has seen this. You get 200 good steps. Accuracy climbs. Then entropy drops off a cliff. The model starts outputting the exact same 7 token prefix for every prompt. Training dies. You back off the learning rate, lower the advantage clip, cross your fingers. It just delays the collapse by 100 steps.

STARE is the first paper that actually explains why this happens, not just proposes another heuristic to hide it. The authors ran first order gradient decomposition on token level GRPO updates and found the mismatch: per token entropy gradient is multiplied by the full trajectory advantage. That means any token that happens to appear in a winning trajectory gets its entropy smashed, regardless of whether that token actually mattered to the outcome.

This is not a minor detail. This was the single largest failure mode for long running RL training. STARE fixes this by reweighting advantages only for tokens that fall above the 90th percentile surprisal for that batch. It adds a closed loop gate that holds entropy within +/- 5% of a target value for the entire run.

Across model scales from 1.5B to 32B and three task families, STARE sustained stable training over 7200 steps. Baseline GRPO died at 1100. On AIME24 and AIME25 accuracy came out 4-8% higher.

If you run LLM RL today, you will replace your GRPO implementation with STARE before the end of next month. There is no tradeoff.

Scalar rewards are obsolete

For two years we have operated on the unexamined assumption that you give the model a 0/1 reward at the end of the trajectory, and the policy will figure out which parts were good.

It never did. Everyone knew credit assignment was broken. Nobody had a good replacement.

Rubric Conditioned Self Distillation changes this. Instead of compressing all feedback into one number, you pass a structured rubric to the teacher model, and get token level guidance on exactly which parts of the rollout satisfied which criteria. No reference answer required. You do not need human written chain of thought. You do not even need a correct final answer to learn from a trajectory.

This beats GRPO by 1 full point across all reasoning benchmarks. More importantly, it stops the model from learning stupid hacks that pass the final check but produce garbage reasoning.

This is not incremental improvement. This is throwing out the core assumption that has defined LLM RL for the last three years.

Training for revision works

All our post training optimizes for single shot answers. All good human engineers revise their work. This misalignment was wasting most of the potential of test time scaling.

REVES fixes this. It takes near miss trajectories that almost succeeded, and splits them into separate revision and verification training examples. The model learns to identify its own mistakes, and learn to correct them, instead of only learning to output perfect answers on the first try.

It gets +6.5 points on LiveCodeBench over standard RL. It solves circle packing with a 4B base model that previously required a 70B model and evolutionary search. Gains hold completely out of distribution on constraint satisfaction puzzles that the model never saw during training.

Before this you could run 32 revisions and get at best 2 points improvement. Now you get consistent linear gains up to 16 revisions.

Rollout generation was 30% slower than it needed to be

Rollout generation is 70-80% of the total runtime cost of RL post training. Everyone knew speculative decoding would help here. Everyone also knew that a fixed drafter stops working after 50 training steps because the target policy changes.

EfficientRollout solves this properly. It runs a 4 bit quantized copy of the policy from 2 steps prior as the drafter. No separate training. No fine tuning. It only runs speculation when the batch is in memory bound regime, which is 85% of the time for RL rollouts. It adapts draft length dynamically based on recent acceptance rate.

Total end to end latency reduction: 12.7%. Zero quality loss. This is a drop in replacement. Nobody will run vanilla autoregressive rollouts six months from now.

Stop building trees, build graphs

Tree based policy optimization was the hot thing three months ago. Everyone was building prefix trees to share rollout computation.

GraphPO kills trees. It merges semantically equivalent reasoning states across all rollouts, not just ones that share a literal prefix. It cuts redundant computation by 40% on reasoning tasks. It gets lower advantage variance because it can compare every path that reached the same state, not just sibling branches in a tree.

This is one of those ideas that everyone will look back on and wonder why nobody did this in 2025. It is strictly better than every existing rollout aggregation method for reasoning tasks. It works with all existing advantage estimators. You do not have to rewrite your whole pipeline to use it.

You can run DiT RL on spot instances

RL for diffusion models was going to require $10M GPU budgets for every model. That was the consensus last month.

Spotlight changes that. The core insight is extremely obvious once someone says it: seed exploration does not need fresh weights. You can run exploration on the previous iteration weights while the current iteration is training. This perfectly fills all the idle time that existed on spot instances.

Spotlight also adds sub second sequence parallelism group reconfiguration that survives preemption. It reaches the same validation score 4x faster than baselines, reducing total cost by 1.4-6.4x while achieving superior image quality.

This is the difference between RL for diffusion being something only OpenAI and Google can run, and something every mid sized team can run next quarter.

The reward was always in the discriminator

Flow matching models have a dirty secret. Even after you train perfectly on the training data, you still have to run RL to get good samples. Everyone thought this was alignment. It was not. It was fixing the loss function mismatch.

Discriminator Guided RL shows you do not need human preferences at all for this step. Train a discriminator on real vs generated samples in a frozen pretrained embedding space. Use the logit as reward. Run one pass of KL regularized RL.

FID on SiT went from 9.38 to 2.62. That is not a small improvement. That is closing almost the entire remaining gap between open models and closed generation models.

The user simulator problem is mostly solved

Turing RL is the first method that produces user simulators that human raters cannot reliably distinguish from real humans. It does this by optimizing directly for indistinguishability, not for matching a single reference response.

This is not just an evaluation trick. Good user simulators are the missing input for training general purpose agent assistants. Until now every agent was trained on static datasets. Now you can run infinite self play against a simulator that behaves like a real human.

Human raters correctly identified simulator responses only 52% of the time. That is effectively indistinguishable.

What still does not work

None of these papers solve multi objective reward tradeoffs properly. Pareto Q Learning with Reward Machines is a good start but it does not scale past 7B parameters yet.

We still do not have good methods for transferring RL gains across base models. Every time you update the base you have to rerun the entire RL run from scratch.

And for all the improvements, RL post training is still 10x more expensive than supervised fine tuning. That gap will close, but it will not disappear.

What happens next

Over the next 90 days every major LLM provider will ship models trained with one or more of these techniques. You will not see it announced. You will just notice that models stop making stupid reasoning mistakes, stop forgetting half way through a task, and stop outputting generic garbage.

This was not one big breakthrough. It was nine separate pieces all landing on the same day. Each one removed one blocker. Together they remove almost all of them.

The hard open problems left are system problems, not algorithm problems. We know how to make RL work now. Now we just have to build it properly.