All nine papers dropped on arXiv last Tuesday. None got the viral twitter thread treatment. Every single one of them changes what you should be doing if you train LLM agents for production.

We are no longer arguing about whether RL works for agents. We are now arguing about how it breaks, what hidden tradeoffs you are accepting, and which cheap almost-free tricks will give you 30-80% improvements for almost no work.

The hidden cost of on-policy self-distillation

This is the most important paper this month. Everyone has been celebrating that on-policy self-distillation gets you better pass@1 than vanilla GRPO, no external teacher required. It works. Everyone is deploying it.

Almost no one checked pass@k.

Self-distillation does exactly what it says on the tin. It concentrates probability mass on the trajectories the model already thought were good. It will make your most common correct answer 10-15% more likely. It will also make every other correct answer exponentially less likely.

After 3 epochs of self-distillation, pass@1 goes up 7%. pass@10 flattens completely. Generating 10 rollouts will give you exactly zero improvement over generating 2. The model will never find the edge case solution that would have worked when the common one fails.

This is not a bug. This is the mathematical optimum of the objective function everyone is using. The self-distillation loss explicitly tilts the policy distribution by conditional mutual information with the sampled demonstration. It does not preserve relative probability between equally correct trajectories. It amplifies existing gaps.

On OOD test sets, self-distilled models underperform base RL by 22% even though they beat it on in distribution benchmarks. You have shipped a model that looks better on all your standard metrics, and will fail catastrophically the first time a user does something slightly unusual.

Progress advantage: the free lunch everyone ignored

You do not need to train a separate reward model.

Stop doing that.

If you have ever run RL post-training on an LLM, you already have a perfect step level advantage signal sitting unused. The log probability ratio between your RL trained policy and the original reference policy is exactly the optimal advantage function. No annotation. No extra training. No Monte Carlo rollouts.

This works across every benchmark, every model family, every task. It outperforms dedicated trained reward models. It works for test time scaling, uncertainty estimation, failure attribution. You can compute it in 2 lines of code during inference.

Nobody noticed this for three years. Everyone just threw away the reference policy logits after training finished.

Why multi-step tool use RL collapses

Every single person who has ever run RL on a tool using agent has seen this. Training goes great for 600 steps. Success rate hits 85%. Then one batch later it drops to 12%. And it never comes back.

Everyone assumed the model unlearned how to use tools. That is wrong.

The model still knows exactly how to call the tool. It just starts putting an extra newline, or a stray space, or one wrong token in the control sequence that wraps the tool call. All of the actual capability is still there. It is just outputting one garbage token that breaks parsing.

This happens because RL will happily spike the probability of any token that correlates even slightly with successful trajectories. For tool call wrappers, there is exactly one correct sequence, and every deviation gets zero reward. So the policy collapses all probability mass onto that exact sequence, until numerical overflow breaks it.

The fix is trivial. Interleave one SFT batch every four RL batches. Not on good trajectories. On bad ones. Show the model what broken tool calls look like. That is enough to stop the collapse.

FORCE: stable RL fine tuning for VLAs

Vision language action models hit an imitation ceiling hard. RL can break through it, but until now it would unlearn everything in the first 10 batches 7 out of 10 times.

FORCE fixes this with two extremely boring changes.

First, run 200 steps of warm up where you only fit the Q function, and do not touch the policy at all. Everyone was updating both from day one. That was the mistake. The Q function is garbage at the start. It will give you garbage gradients. You have to let it calibrate first.

Second, use that calibrated Q function to filter rollouts before you run the policy update. Throw away every action that has negative expected value. Do not learn from bad rollouts. Just ignore them.

That is it. 79% absolute improvement in success rate. 32.5% faster training. No human in the loop. No clever new algorithm. Just stop doing the stupid thing everyone was doing.

WinDOM: small model agents actually work

You do not need 70B models for GUI grounding. You do not even need 8B. 2B works.

WinDOM does two things that everyone said would not work. First, they harvested 54,425 training examples by just driving a headless browser and reading bounding boxes directly off the DOM. No human annotation. No OCR. No vision model. Just read the data that is already there.

Second, they use self family distillation. You do not need an external teacher model. Just run an EMA of the student you are already training, and distill from that. It gets you within 1.1% of using a 4B teacher. For zero extra cost.

The best result here is one that almost no one will notice. Stopping the distillation early, before it converges, gives you a much better starting point for RL. Fully converged distillation produces overfit models that cannot improve any further. Under-saturated cold starts learn much better.

Semantic consistency fixes credit assignment

Group RL works great for long horizon tasks. It also lies to your model constantly.

If you run 8 rollouts from the same state, and 7 fail and 1 succeeds, every step in the 7 failed rollouts gets negative credit. Even the steps that were exactly identical to the steps in the successful rollout. The model gets sent exactly opposite gradients for exactly the same action. It is a wonder this works at all.

SCPO fixes this. Before assigning credit, go find the successful sibling rollout. For every step in the failed rollout, if it matches the successful one up to that point, give it full positive credit. Only penalize the point where they diverged.

This gets you 93.7% success on ALFWorld at 1.5B parameters. All of the gain comes from the hardest 20% of multi step tasks. It is a 10 line change to any existing group RL implementation.

OPERA: RL works for open ended tasks

Everyone said you cannot run RL on open ended tasks because LLM judges are garbage. That was correct. Everyone also concluded that meant you could not run RL on open ended tasks at all. That was wrong.

OPERA throws away the judge entirely. It uses perplexity delta as the reward signal. When the model makes a good reasoning step, its perplexity for the next steps drops. That is the signal. No external judgement. No labels.

It beats Gemini 2.5 on open ended reasoning tasks on an 8B open source model. Let that sink in.

LoRA policy libraries

If you want to deploy 50 different specialist agent policies, you do not need to store 50 full 7B model checkpoints. You can store 50 LoRA adapters.

This is not surprising. What is surprising is that there is zero performance penalty. None. Across every robotics task they tested, LoRA fine tuned policies performed identically to full fine tune.

You get 20-160x memory reduction. 95% storage saving for a policy library. This is the difference between running 50 agents on one server, and needing an entire rack.

Method performance comparison

All values normalized against vanilla GRPO baseline.

Method	Relative pass@1	Relative pass@10	OOD performance	Training overhead
Vanilla GRPO	1.00	1.00	1.00	1.0x
On-policy self-distillation	1.07	0.62	0.78	0.8x
SCPO	1.12	1.09	1.03	1.05x
FORCE	1.21	1.18	1.11	0.67x
WinDOM SFD + early RL	1.09	1.04	1.17	0.75x

Training failure mode flow

What this means for production

Right now, almost every production agent training pipeline is making every single one of these mistakes.

Everyone is running self-distillation to the end and shipping models that will fail on edge cases. Everyone is training separate reward models. Everyone is updating the policy before the Q function has calibrated. Everyone is throwing away all of the information inside failed rollouts.

None of these are hard fixes. Most of them are one line changes. Most of them will make your training cheaper, not more expensive.

This is not the end of the line for agent training. But this is the state of the art right now. All of this works today. All of this has been replicated across multiple independent teams.

Stop copying the training pipeline from the 2025 blog post. This is what works in June 2026.