This is not another roundup of incremental RL papers.

Every one of these works dropped on arXiv in the last 72 hours. Taken together they do not just move benchmark numbers. They answer three questions every ML engineer has been asking about RL for the last three years:

1. Do we have to run full RL on every new 700B model we train?
2. When will RL actually beat hand written heuristics in production?
3. Are all the nice theoretical RL results still based on assumptions that never hold in the real world?

The end of per-model RL fine tuning

This is the biggest result this batch. Direct-OPD changes everything about how we will run RL on foundation models.

Right now every time you release a new base model you burn tens of thousands of GPU hours running PPO, DPO or RLVR all over again from scratch. Nobody has been able to reuse RL work across model sizes. Distilling a fine tuned small model just gives you a bad copy of the small model, not an improved large one.

Direct-OPD does not distill the final policy. It distills the delta. You run RL once on a cheap 1.7B model. You record exactly which actions RL made that model more or less likely to take. Then you apply that exact delta to any larger model. No reward model. No rollouts on the target model. No RL training loop on the big model at all.

The results hold perfectly across scale. They took the RL delta from a 1.7B Qwen run, applied it to the same 1.7B model and got exactly the same result as running full RL. Then they applied exactly the same delta to Qwen3-7B, Qwen3-14B and Qwen3-72B. Every single one improved by almost exactly the same margin.

This works because RL is not teaching the model new facts. It is teaching the model which direction to adjust its existing distribution. That direction transfers perfectly across scale.

You can now run RL once on a tiny model for 100 dollars, and apply the result to every model you will ever train for the next year. That removes the single largest cost bottleneck for RL on foundation models.

Direct-OPD benchmark results

CompactionRL fixes the long horizon agent context wall

Everyone building LLM agents has hit this wall. You run an agent for 200 steps, it runs out of context window, and forgets what it was doing.

The standard fix is to summarize old context. Nobody has been able to train RL on summarized trajectories before. Every time you compress context you introduce bias that breaks generalized advantage estimation completely.

CompactionRL solves this. It jointly trains the agent to do the task and to generate its own context summaries, with normalized loss that exactly cancels out the compression bias during GAE calculation.

This is already shipping. Zhipu ran this on GLM-4.5 and got 7 points on SWE-bench Verified. They are running it right now for the GLM-5.2 750B training run.

Model	Baseline SWE-bench	CompactionRL SWE-bench	Absolute Gain
GLM-4.5-Air 106B	59.8%	66.8%	+7.0
GLM-4.7-Flash 30B	50.5%	56.0%	+5.5

The hard boundary for RL in systems engineering

This paper is the most useful practical result for production engineers in this entire batch.

Everyone argues about whether RL works for inference batching. Nobody has ever run a clean controlled test that tells you when you should use it and when you should not.

They tested PPO and REINFORCE agents against every standard heuristic across single and multi GPU serving. The result is extremely clear.

On a single GPU. RL does not win. A well tuned static batch size is within 1% of optimal. You will never justify the engineering cost of running an RL agent here.

On heterogeneous multi GPU routing. RL destroys every heuristic. It discovered a policy that segregates long and short requests across GPUs, completely eliminating head of line blocking. 48% better throughput than shortest queue. 3.5x better than round robin.

This is the rule you should write down: RL beats heuristics only when the problem requires coordinating multiple independent resources. For any single resource scheduling problem, just tune the heuristic. Do not waste your time.

RL policy batching decision flow

Theory finally catches up: FORE removes Bellman completeness

For twenty years every offline RL paper came with a giant asterisk. All the convergence proofs required Bellman completeness. An assumption that is never, ever true for any real world function approximator. Everyone knew this. Everyone ignored it.

FORE changes this. They proved that you only need one condition for fitted off policy evaluation to converge: the occupancy ratio must be representable in your hypothesis class. That is it. No Bellman completeness. No operator stability. No value function realizability.

This is not a minor tweak. This is the first time we have a theoretical foundation for offline RL that actually matches the conditions we run it under in practice.

Trajectory neglect and STAPO

Trajectory neglect is the silent failure mode of every LLM agent. Half way through a 50 step task the agent will just forget the original goal, and start doing something completely unrelated. Nobody had a good measurement for this, nobody had a fix.

STAPO introduces normalized entropy. Instead of measuring absolute entropy at each step, it measures how much the agent's action distribution deviates from its own average behaviour for that state. Outlier steps are almost always the ones where the agent drifted off task.

They use this signal to apply targeted gradient updates only to the bad steps, instead of updating the entire trajectory. On ALFWorld this reduced trajectory neglect events by 72% and improved final success rate by 11 points.

Real world deployments that are not benchmarks

This is the quietest shift in this batch. Half these papers are not running on Atari or Mujoco. They are running on real engineering problems.

RL-Ballast is deployed on commercial container ships right now. It does ballast water routing and clog prediction. It beats Dijkstra by 32% on decision steps when pipes get blocked. It can diagnose clogs with no extra flow sensors.

The relational multi agent RL for railway pricing is running in trial on the UK rail network.

None of these papers have fancy marketing. None of them have twitter threads. They just work.

The reward design bottleneck is still unsolved

For all the progress this month, reward design is still the hardest unsolved problem in RL.

CuRe for video captioning is a good step forward. It breaks captions down into atomic claims and verifies each one individually. This reduces reward hacking substantially. But it still does not solve the core problem: we still do not know how to write a reward function for any open ended task.

Every single successful RL deployment this month works on tasks with a clear, verifiable, binary success condition. That is not a coincidence.

Closing observation

For the last five years RL research has been split between two groups. Theorists proving results under impossible assumptions. Practitioners hacking together things that worked but nobody could explain.

That split is ending. This month we got practical methods that work at foundation model scale. We got theory that actually applies to the systems we run. And most importantly, we got RL deployed solving boring, real, important engineering problems that nothing else could solve.

RL stopped being a research toy this month. It is now just another engineering tool. One that you will be expected to know when to use, and when not to.