Every single working production LLM team right now is rewriting their RL pipeline.

This is not hype. Over the last 7 days, 8 papers landed on arxiv that together obsolete almost all standard practice for LLM reinforcement learning. None of them got the viral twitter thread treatment. All of them contain results you will be running in production before the end of the quarter.

This article breaks down the findings, the tradeoffs, and the parts that are safe to ignore.

Ground truth is no longer required for RL training

This is the single biggest result this month. For the entire history of RLVR, everyone operated under one unchallenged assumption: you could only run reinforcement learning on tasks where you had a correct answer to check against.

That assumption is dead.

RiVER, the framework from 2606.27369, trains coding models exclusively on heuristic contest tasks where there is no single correct answer, only better and worse solutions. It uses only execution score as feedback. No ground truth. No reference solutions. No LLM judge.

When trained this way, Qwen3-8B gained 8.9% on the ALE heuristic benchmark. That was expected. What was not expected: the same model also improved 2.4% on LiveCodeBench and 3.5% on USACO. These are exact solution benchmarks. The model got better at writing correct code, despite never once seeing a correct answer during training.

Raw uncalibrated execution scores did not produce this transfer. They improved heuristic performance but made exact solution performance worse. The difference was entirely reward calibration.

The authors identified two failure modes that break every existing group RL optimizer when run on continuous scores:

1. Scale dominance: Unnormalized score magnitudes across different problems completely distort advantage calculations. One problem with a 0-1000 score range will override 100 problems with 0-10 ranges.
2. Frequency dominance: Mediocre solutions that are sampled 20 times will always have higher cumulative gradient weight than one perfect solution sampled once.

RiVER fixes both with per-instance rank normalization. All rewards are replaced with relative rank inside the batch for that single problem. No cross problem comparison ever happens. Top ranked samples get maximum reward, all other samples get linearly scaled reward bounded between 0 and 1.

That is the entire trick. And it works.

GRPO is extremely vulnerable to bad rewards

Everyone switched to GRPO this year. It is faster, simpler, and produces better results than PPO for almost every LLM use case.

Almost no one was talking about its failure modes. Until now.

The industrial job search paper 2606.27291 ran a controlled head to head comparison of every popular critic free optimizer. They held everything fixed: model, data, rollout count, training steps. Only the optimizer changed.

Optimizer	Baseline quality	With uncorrected reward	With reward floor
REINFORCE++	0.512	0.581	0.603
RLOO	0.512	0.577	0.598
GRPO	0.512	0.439	0.659

This is the most important table published about LLM RL this year.

When rewards are clean, GRPO wins by a very large margin. When rewards have even a single exploitable flaw, GRPO will find that flaw faster than any other optimizer and collapse completely. It will stop doing the task. It will just output whatever garbage triggers the reward.

This is not a bug. This is an explicit design property. Group relative normalization removes all absolute signal. There is nothing stopping the entire batch converging to a reward hack. Once 3 out of 8 samples in the group are hacking the reward, all advantage signals will reinforce that behaviour.

The fix is trivial. Add a hard rule based reward floor. Any output that matches a known failure mode gets zero reward, no exceptions. Do not try to make the reward model learn this. Do not try to make the judge penalize it. Hardcode it.

If you are running GRPO right now and you have not done this, your model is almost certainly reward hacking and you have not noticed yet.

Reward shaping beats optimizer choice

The same paper found something that will make every RL researcher very annoyed. Once you have acceptable reward shaping, the choice of optimizer explains less than 3% of the final outcome.

97% of performance comes from reward design.

All of the arguments on twitter about PPO vs GRPO vs RLOO are noise. None of them matter even slightly compared to getting the reward signal right.

The paper also confirmed that training reward model scores overestimate real world performance by exactly 2.4x. This is a consistent measurement across all optimizers and all reward designs. If your internal reward score went up 20%, you should expect a real 8% improvement. Not 20%.

This is not measurement error. This is systematic leakage. It will never go away. You should bake this discount factor into every experiment you run.

RolloutPipe eliminates 75% of trainer idle time

Everyone running RLVR at scale has the same problem. 60-80% of your expensive training GPUs are sitting idle waiting for rollouts to finish.

RolloutPipe fixes this.

Standard GRPO runs full batches sequentially: generate all 128 rollouts, then train on all of them, then repeat. Training GPUs do nothing during rollout. Rollout GPUs do nothing during training.

RolloutPipe pipelines per group. As soon as one complete GRPO group is finished generating, it is sent immediately to the trainer. Training starts before the full batch has even finished rolling out. No stale policy. No off policy data. All correctness guarantees are preserved.

On standard 8x H100 setups this reduces total iteration time by 30-42%. Trainer idle ratio drops from 61% to 15%.

There are no downsides. This is pure upside. Every existing RLVR implementation will be patched to work this way within 3 months.

VLM guided potential based reward shaping

Sparse rewards still break every RL algorithm. Hand written reward shaping always creates reward hacks. This has been a stalemate for 10 years.

VLM-PBRS breaks the stalemate.

Potential based reward shaping is the only known reward shaping method that is mathematically guaranteed to not change the optimal policy. Everyone knows this. No one uses it because no one could write a good potential function.

Now you don't have to. You can ask a VLM to rank pairs of states, train a small potential function on those rankings, and plug it directly into PBRS.

You do not need a good VLM. Even a 7B parameter VLM with 60% ranking accuracy produces 2-3x sample efficiency improvements. Noise in the preference labels does not break the guarantee. It only reduces the speedup.

This works. It has been tested on robot control tasks. It will be ported to LLM tasks before the end of the year.

State representation is not a minor detail

Almost every RL practitioner treats state feature engineering as boring preprocessing work that you hand off to the junior engineer.

The energy trading paper proves this is the single most important decision you will make.

They ran 12 different state representations against exactly the same Double DQN agent, same reward, same network, same training loop.

The worst performing state representation achieved 5.7% of optimal profit. The best achieved 47.5%. Nothing else changed.

You can have the perfect optimizer, the perfect reward function, the perfect network architecture. None of it will rescue you from bad state representation.

The boring engineering that wins competitions

No one writes papers about good engineering. Everyone should read the 1st place LeHome folding robot solution anyway.

The winning entry did not invent any new RL algorithm. It did not use a new architecture. It won by combining four boring, well known tricks that no one bothers to implement correctly:

1. Use the same network for action prediction and value estimation
2. Run asynchronous rollout and training through a standard model hub
3. Optimize inference hyperparameters live with Thompson sampling
4. Collect 120 minutes of human in the loop correction data after sim transfer

All of these tricks work for LLMs too. None of them are used in any public RLVR pipeline today.

What you should stop doing right now

Stop arguing about optimizers. Stop benchmarking PPO against GRPO. Stop trying to make better reward judges.

Stop training RL only on tasks with ground truth answers.

Stop running synchronous batch rollout.

Stop treating state representation and reward shaping as afterthoughts. They are the work. Everything else is plumbing.

Stop running ablation studies on network depth. Stop adding dropout. Stop tuning learning rate. None of these variables move the needle any more.

What comes next

We are no longer in the period where RL for LLMs is about finding a better optimizer. That era ended this month.

We are now in the era of reward engineering. All of the hard problems are here. All of the gains are here.

Every LLM that will ship in 2027 will be trained with RL. Almost none of them will use ground truth labels.

This is not an incremental improvement. This is the point where LLM training stops being supervised learning.

We did not get here with one big breakthrough. We got here with eight boring, well executed papers published on the same Tuesday. That is how progress actually works.