Every production team running RL fine-tuning right now is operating on a playbook written 12 months ago. That playbook is wrong.

Over the last week four independent papers landed on arXiv that collectively invalidate almost every standard assumption people have been building RLHF pipelines around. None of them got the viral twitter thread treatment. All of them will change what you run in production before the end of the year.

We have been wasting compute pretraining Q functions

This is the most embarrassing result published in this field in years. For 18 months every guide, every reference implementation, every production RL pipeline has started RL fine-tuning by first pretraining the critic / Q function on 100k+ offline rollouts from the base model. Everyone did this. Everyone said this was required for stable training.

It turns out you get better results if you just initialize the Q function randomly.

The authors of Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning? ran the control experiment nobody bothered to run for two years. They tested standard PPO, GRPO and DPO pipelines across 11 benchmarks, and found that in 9 out of 11 cases a randomly initialized critic reached higher final performance, converged faster, and had lower gradient variance than one that had been pretrained for 10 epochs on offline data.

The reason is obvious once someone points it out. The Q function you pretrain is an accurate estimator for the base policy. It is not an accurate estimator for the policy you will end up with after 1000 gradient steps of online fine tuning. Worse, it is actively misleading. The pretrained critic will incorrectly penalize every good new behaviour the policy discovers that was not present in the offline dataset. This creates a permanent drag on improvement for the entire fine tuning run.

They did find one case where Q pretraining helped: when you were only going to run less than 50 fine tuning steps. For any run longer than that, random initialization wins.

The paper proposes a minor improvement called Initialization via Policy Ensemble which gives another 26% average gain. That is nice. The actual result here is that everyone has been burning millions of dollars of GPU time on a step that actively makes their models worse.

GRPO is concentrating your model into a local minimum

Group Relative Policy Optimization ate the world over the last six months. It is simpler than PPO, more stable, trains faster, almost always gets higher Pass@1. Every major model release since Llama 3 has used GRPO.

Nobody was checking Pass@16.

ReCo: Reweighting GRPO Against Distributional Concentration documents the silent failure mode that every GRPO fine tuned model has right now. While Pass@1 goes up, Pass@k for k>2 collapses. By the end of a standard GRPO run the model will only produce 3 or 4 distinct reasoning paths for any given problem, even if the base model could generate 20+ correct ones.

This is not just an academic metric. This is why your fine tuned model feels rigid. This is why it will never find the clever edge case solution that the raw base model would occasionally produce. This is why agent loops built on fine tuned models perform worse than ones running base models.

The paper identifies two separate failure modes in the GRPO update rule:

1. At the response level: if one good response happens to be sampled 3 times in a group of 8, it will contribute 3x the gradient of every other response. Good common responses rapidly take over the entire distribution.
2. At the token level: the importance weight multiplier in GRPO exponentially reinforces tokens that are already becoming more probable. Once the model starts picking a token, it very quickly stops considering any alternative.

ReCo fixes both with two trivial changes. Normalize response gradients by their expected frequency in the group. Replace the token importance ratio with a variance scaled weight that applies larger updates to tokens where the model still has meaningful uncertainty.

GRPO vs ReCo benchmark results

Notice what is happening here. ReCo matches GRPO exactly on Pass@1. It completely restores the base model's distribution coverage for higher k. This is the best kind of improvement. There is no tradeoff. You can drop this into your existing GRPO pipeline today and get strictly better models.

Test time RL no longer needs reward models

Everyone has operated under the axiom that RL for LLMs requires an external reward model. You need a judge. You need labeled data. You need something outside the policy to tell it what is good.

SERPO breaks that axiom completely.

SERPO: Self-Evolving Rubric Policy Optimization demonstrates that a model can align itself at test time, with no external feedback, no reward model, no human labels.

SERPO execution loop

This is a closed loop. The model generates the candidates, sorts them, writes the grading criteria, and updates itself. There is nothing outside.

And it works. Across 6 benchmarks SERPO improves base model performance by an average of 8 points, and up to 20.6 points on HealthBench. It continues improving for hundreds of steps. It transfers across benchmarks. It does not collapse into repetition.

This is not just a better fine tuning method. This changes the entire alignment model. You do not need to bake all alignment into the model before deployment. Models can correct themselves, improve themselves, and adapt to new standards while running.

Agents learn skills by just writing them down

SkillRise is the cleanest agent RL architecture anyone has proposed to date.

For the last year all agent skill learning systems worked the same way. Run episodes. After the fact run a separate extraction step to pull out skills. Store them in a vector database. Retrieve them at the start of future episodes.

This pipeline is slow, brittle, and almost never actually transfers across tasks.

SkillRise throws all of that away. There is no extraction step. No vector database. No separate retrieval model. There is only one policy.

At every step the policy gets two outputs. It writes the action to take for the current task. It also writes one paragraph appended to a running skill document. That skill document is passed unmodified into the context window for every subsequent task.

Credit assignment is trivial. The action gets reward from the current task outcome. The skill paragraph gets discounted reward from all future tasks.

That is the entire system.

Method	ALFWorld Pass@1	WebShop Pass@1	ScienceWorld Pass@1
Base LLM	61.2	57.8	42.1
ReAct	70.4	63.2	48.7
SkillMem	74.9	67.1	52.3
SkillRise	77.2	69.4	54.6

The policy learns on its own what is worth writing down. It learns to refine previous notes. It learns to ignore bad notes. It does all of this without any supervision about what a skill even is.

Most remarkably, performance continues to improve the more consecutive tasks the agent runs. It gets better with experience, exactly the way a human would.

None of this was predicted 6 months ago

The pace of progress right now is absurd.

Six months ago everyone agreed that:

• Q pretraining was required for stable RL
• GRPO was strictly better than every alternative
• You could not do RL without an external reward model
• Skill extraction required dedicated pipeline stages

All of that is gone. All of it was overturned in one week of papers.

None of these results relied on larger models, more data, or new hardware. Every single one was just someone finally running the obvious control experiment that everyone else had assumed was not worth running.

What you should change this week

Stop pretraining your Q function. Just initialize it randomly. If you are running GRPO, implement ReCo right now. There is no downside. Test SERPO on your internal tasks. This is not a toy. Start experimenting with SkillRise for agent loops.

None of these papers require new hardware. None of them require more compute. All of them are algorithmic improvements you can drop into existing pipelines today.

Closing observation

The strangest pattern across all four papers is that every single improvement came from removing something. We stopped doing Q pretraining. We removed the bias from GRPO. We removed the external reward model. We removed the entire skill extraction pipeline.

We have spent three years adding complexity to RL for LLMs. It turns out almost all of that complexity was making things worse.

We are only just starting to figure out how this actually works.