Every single RL paper dropped on arxiv this week is saying the same thing, and almost no one has noticed yet.

We lost the plot on reward optimization. For three years everyone was just trying to make the reward number go up. We built better optimizers, better reward models, longer rollouts. And it worked. Until it didn't.

Every production RL pipeline now has exactly the same failure mode: you crank the reward, you get higher benchmark scores, and every other desirable property of the model rots away. Intelligibility breaks. Readability vanishes. Temporal consistency dies. Safety degrades.

All five papers released 26 June 2026 are attacking exactly this problem. None of them are trying to get higher maximum reward. All of them are fixing the structural mismatch between how we measure reward, and how the model actually generates output.

That is the next phase of RL. That is what will actually get aligned models into production.

The core failure mode of modern RL

This is the universal problem across every domain. You have a dense generation process: speech produces 50 frames per second, video produces 8 frames per second, reasoning produces one token at a time, agents produce one turn per interaction.

And you have one sparse reward at the end.

When you backpropagate that single number through 1000 steps of generation, the gradient signal dissolves into noise. The model finds whatever local shortcut correlates most strongly with the final reward. It does not learn the behaviour you wanted. It learns to hack the reward.

This is not a bug in your optimizer. This is not a bad reward model. This is a structural scale mismatch.

Every paper this week is a different solution to this exact mismatch.

HPRO: Isolate reward signals before you optimize

HPRO for emotional TTS is the cleanest demonstration of this principle you will ever see.

Standard preference optimization for TTS takes the full speech latent space and shoves it straight into a reward model. This always produces the same failure: you turn up the emotion reward, and the speech stops being intelligible. Content and emotion are encoded in the same latent space. Gradients for emotion overwrite gradients for content.

The HPRO authors did not build a better reward model. They built a reward model that first splits the signal into two completely separate token spaces: one for semantic content, one for emotional style.

Only after this structural separation do they apply preference optimization. They optimize emotion on the emotion tokens only. They never touch the content tokens.

No gradient clipping. No penalty terms. No clever loss functions. They just stopped sending conflicting gradients to the same place.

Results are unambiguous. HPRO gets 38% higher preference score for emotion, while measurable word error rate actually improves by 1.2% relative to SFT. No other method has ever done both at the same time.

Method	Emotional Preference	Word Error Rate
Base SFT	50.0%	3.12%
Standard DPO	67.1%	4.78%
GRPO	71.4%	5.21%
HPRO	83.2%	3.08%

That table should make you stop and re-evaluate everything you thought you knew about RLHF. You do not have to trade off properties. You just have to stop optimizing them on top of each other.

Tandem RL: Reward compatibility not just capability

Everyone is excited that RL with verifiable rewards gets 90% on AMC 12. No one is talking about the fact that the output is unreadable garbage.

RLVR trained models do not reason like humans. They do not even reason like other LLMs. They have found alien reasoning paths that correlate perfectly with correct final answers, and make zero sense to any other observer.

Tandem RL fixes this without giving up any performance.

You run two models during training:

1. Senior: the strong model being trained
2. Junior: a frozen weaker baseline model

They alternate writing every other step of the chain of thought. The entire team only gets reward if the final answer is correct.

That is the entire trick.

The senior can no longer use idiosyncratic reasoning shortcuts. If it writes a step the junior cannot follow, the junior will make a mistake on the next turn and the whole rollout gets zero reward.

On Qwen3-4B, TRL matches vanilla GRPO exactly on final math accuracy. At the same time, junior model ability to continue a half finished chain of thought improves by 41%. Human readability scores go up 27%.

There was no reward term for readability. There was no reward term for compatibility. It emerged automatically, because the reward structure required it.

KCPR: Stop treating reward and punishment as opposites

Every existing RPRL implementation runs two completely separate policies. One policy chases reward. One policy avoids punishment. They never talk to each other.

This works badly. The reward policy will walk straight into danger to get one extra point. The punishment policy will freeze and do nothing rather than accept any risk.

KCPR couples the two policies explicitly. Each policy is used as the prior for the other.

When the reward policy considers an action, it is penalized by how far that action deviates from what the punishment policy would have done. When the punishment policy considers an action, it is penalized by how far that action deviates from what the reward policy would have done.

They do not pull in opposite directions. They negotiate.

On robotic navigation tasks klDMP, the reference implementation, had zero collision events across 1200 test rollouts. DQN had 112. SoftDMP had 76. Task completion rate was within 1% of the best pure reward baseline.

This is not a tradeoff. This is coordination.

TempAct: Reward at the granularity you generate

Autoregressive video generation has exactly the scale mismatch problem we opened with. You get one reward at the end of a 120 frame video. You backpropagate that reward through every denoising step.

The model learns that if it makes the final frame look good, it gets high reward. It does not learn anything about what should happen in between.

TempAct does not use one global reward. It has three separate reward signals, applied exactly at the granularity they apply:

1. Transition reward: applied every 8 frames, checks that the prompt boundary was followed correctly
2. Plan reward: applied once per prompt sequence, checks that events happened in the correct order
3. Full video reward: applied at the end, checks overall quality and aesthetic

Crucially, each reward signal is only ever backpropagated to the parts of the generation that were actually responsible for it. No credit assignment across boundaries.

This is such an obvious idea it is embarrassing no one did this five years ago. You would never grade an essay by reading the last line and then giving the same mark to every sentence. That is exactly what every RL pipeline was doing until last week.

ATOD: Know when to imitate and when to explore

ATOD solves the other universal failure mode of agent training.

You have two options:

1. Distill from a teacher. Fast start. Hits a hard ceiling exactly at the teacher's performance.
2. Train with RL. Slow start. Can exceed the teacher. Almost always dies horribly before getting there.

ATOD does not pick one. It anneals smoothly between them over the course of training.

First 20% of training: 100% on policy distillation. Zero RL. Copy the teacher exactly.

Then every training step you very slowly turn up the RL weight and turn down the distillation weight. By 70% of training it is 100% RL.

Additionally they reweight loss per turn, not per trajectory. Turns where the teacher and student disagree get higher weight. Turns where they agree get almost no weight.

On ALFWorld, WebShop and SearchQA this beats pure OPD by 3.0 points, beats pure GRPO by 23.6 points, and beats the original teacher model by 2.2 points.

That last number is the important one. This is the first reliable method we have that consistently produces a student model better than the teacher it was distilled from.

The unifying principle across all five papers

None of these papers use a new optimizer. None of them use a bigger model. None of them get more training data.

Every single improvement comes from changing the structure of the reward signal, not the magnitude.

All of them follow exactly three rules:

1. Never apply a reward signal at a different granularity than the process it is measuring
2. Never optimize for multiple objectives on the same latent space
3. Reward the interaction you actually want, not the final outcome

For three years the entire field was trying to make reward stronger. We made it sharper. We made it denser. We cranked the learning rate.

All we had to do was put the reward in the right place.

What this means for production alignment

This is the end of the one big reward model era.

You will never again build a single model that looks at the full output and gives one score. That approach has hit its limit. It will never produce aligned, reliable, usable models.

Future RL pipelines will have structured reward. They will split latent spaces. They will apply rewards at multiple levels. They will reward interaction between components. They will not maximize a single number.

This is not a small incremental improvement. This is a model shift. This is the point where RL stops being a hack that makes benchmarks go up, and starts being an engineering discipline that produces predictable behaviour.

Open questions

No one has yet combined these techniques. We do not know what happens when you run tandem RL inside a hierarchical reward framework. We do not know what happens when you anneal distillation on top of coupled reward punishment policies.

All of these papers were run on small models. No one has run any of this at 70B parameters. No one has run this on production traffic.

But the direction is unambiguous. Reward optimization is no longer about maximization. It is about structure.

That is the thing everyone missed this week. And it will be the foundation of every aligned model shipped over the next two years.