The core news

This is not an incremental week. Every single major assumption people have been running RLHF on for the last three years got challenged, corrected, or replaced with a strictly better alternative in one batch of arXiv submissions.

You can stop using PPO clip tomorrow. You can stop training on scalar rewards next week. You can stop wasting compute fully fine tuning small models for reasoning. All of this has working published benchmarks right now.

None of this requires 70B models. All results run on 1B and 1.5B parameters.

PPO clip is obsolete. Meet OR.

For five years every RLHF pipeline ran clipped surrogate objectives. Everyone knew the clip threshold was a terrible hyperparameter. Everyone knew the derivative dies hard at the boundary. No one had a good replacement.

Until now.

OR, Output Reset, is a smooth one sided saturation rule that replaces the clip operator entirely. Instead of flatlining the gradient once you cross the margin, OR smoothly decays residual gradient to zero. There is no hard edge. There is no dead gradient region.

The authors ran head to head matches on Llama 3.2 1B Instruct on hh-rlhf. Same reward model. Same rollout batch. Same seed set.

That is +0.305 mean reward. For a one line change to your loss function. No other changes.

This is not a small gain. This is the largest single improvement to base PPO ever published for LLM fine tuning.

GRPO also gets the OR treatment. It did not deliver higher mean score at group size 2, but it eliminated almost all run to run variance. For anyone that has ever run GRPO three times and got three completely different models, this alone is reason to switch.

There is one catch. OR does not reduce log ratio drift. You still need gradient clipping. You still need rollout freshness. Everything else stays exactly the same.

You should implement this this week.

Stop aggregating log ratios

The multi agent PPO paper is the most important practical result in this entire batch. Almost everyone has been doing this wrong.

When you run cooperative policy optimization you have two choices for aggregation:

1. Which agents do you sum over when calculating advantage?
2. Which agents do you multiply over when calculating importance weight ratios?

Every existing method picked arbitrary combinations. This paper proves something extremely clean: these two operations have completely different variance properties.

Advantage is a sum. Variance adds linearly. There is an optimal window size where bias and variance cross. You absolutely should aggregate neighbors here.

Ratio is a product. Variance grows exponentially. There is no optimum. There is only worse. You should never aggregate anything here. Ever.

Aggregation location	Variance behaviour	Optimal support size
Advantage	Additive linear	Finite, matches coupling neighbourhood
Log ratio	Multiplicative exponential	1. Always 1.

This is not an opinion. This is a proof.

Every existing multi agent PPO variant was doing this backwards. MAPPO aggregates advantage correctly but also aggregates ratios. IPPO does neither. HAPPO aggregates ratios and not advantage. All of them leave performance on the table.

The correct rule is now unambiguous: aggregate in the advantage. Never aggregate in the ratio.

Reward scalars are a bad idea

LLM as a Judge was always a stupid hack. You take a 1B parameter model that wrote three paragraphs of detailed feedback, throw all of it away, and keep one number between 0 and 1.

Experiential Learning fixes this. Instead of a scalar reward, you keep the coach feedback. You inject that feedback directly into the policy context window during distillation. The model does not learn to maximise a number. It learns to internalise the actual critique.

Results are consistent across every tested task. EL beats standard RLHF on held out performance. It generalises far better out of distribution. It almost completely eliminates reward hacking.

This is not RL anymore. This is not credit assignment. This is direct experiential transfer.

You do not need an external reward model. You can run this with the same model acting as coach for itself. It still works.

This is the end of scalar reward RLHF. It will be gone within 12 months.

RAG is just nearest neighbour causal matching

The RAG policy paper connects two fields that everyone suspected were the same thing but no one had ever formalised.

When you run vector search to select actions, you are doing exact nearest neighbour matching from causal inference. The regret decomposes exactly the same way. The error bounds apply exactly the same way.

This means every single thing we know about causal matching can now be imported directly into RAG policy systems. You can bound regret. You can detect confounding. You can calculate sample requirements.

No one had done this before. Everyone was treating RAG as an engineering trick. It is not. It is a formal causal estimator.

This also means most production RAG systems are running with provably suboptimal k values. Most people run k=3 or k=4. The causal bound says optimal k scales with the fourth root of dataset size. For most production corpuses that means k=12 to 16.

MADA-RL: 2% accuracy for 1/16th the parameters

If you are running small models, stop full fine tuning them.

MADA-RL splits a single compact model into generator and critic roles using LoRA adapters. Only 0.3% of parameters are trained. It uses a counterfactual advantage that trains the critic to beat the generator consensus instead of just matching ground truth.

On DeepSeek 1.5B this delivered +2.0 percentage points on math reasoning. That is the same gain you get from full fine tuning, using 16 times fewer trainable parameters.

At inference time you just run the generator. There is zero overhead. No multi agent runtime. Nothing changes for end users.

This is the single most efficient post training method for small models published to date.

Bellman was not magic

The generalised Bellman paper answers a question almost no one asks: why does the Bellman equation work at all?

It turns out Bellman recurrence is not a fundamental law. It is an emergent property that only holds when three separate conditions are all satisfied. Break any one, and the equation stops working.

This explains almost every failure mode in deep RL. Every time you see value function collapse, every time Bellman backups diverge, you broke one of the three conditions.

Most importantly: you can fix this. You do not have to add more layers. You do not have to tune learning rate. You can explicitly check which condition failed, and repair it.

This is the first useful theoretical result about Bellman operators produced in ten years.

Cramér geometry fixes distributional RL

For anyone working on distributional critics, the contraction proof under Cramér distance removes the last major theoretical blocker for production deployment.

Prior work could only prove contraction under restrictive bounded reward assumptions that never hold for LLM reward models. This proof only requires a uniform first moment condition, which is satisfied for every practical reward signal used today.

This means you can stop clipping rewards. You can stop running value function rescaling hacks. The Bellman operator will contract. It will converge. It will not blow up.

What does not work

None of these papers are silver bullets.

OR does not fix log ratio drift. It does not make bad reward models good. It just makes the optimization step not broken.

Experiential Learning still requires the coach to actually be able to give good feedback. Garbage in garbage out still applies.

MADA-RL does not beat much larger models. It just makes small models as good as they can possibly be for the compute you spent.

None of this will save you from a bad dataset.

Closing

This is what progress looks like. For three years RLHF was a pile of heuristics that mostly worked and no one understood. That period just ended.

We now have clean proofs. We have head to head benchmarks. We have actionable changes that deliver measurable gains for zero additional compute.

You do not have to wait for any vendor to ship this. You can implement every single one of these changes this month.

All the papers are linked. Go read them.