Every single one of these papers solves a problem you have hit if you have ever tried to train or align a reasoning LLM for production. None of this is benchmark wank. Every result comes with actionable implementation steps, every failure mode described is one you have wasted weeks debugging.

β-OPSD fixes the thing that broke your self-distillation runs

Everyone who tried On-Policy Self Distillation in the last 6 months knows the joke: it works great once every 12 runs, no one knows why, and the person who got it working left the company.

This paper did the obvious thing no one bothered to do: they proved vanilla OPSD is just a special case of policy optimization with β fixed at exactly 1. That was not a design choice. That was an accident.

β controls the KL penalty anchoring the student to the reference policy. At β=1 you get exactly the original OPSD update rule. At β=0 you get pure imitation of the teacher. Every value in between trades stability for reasoning gain.

You do not run RL for this. You do not need any new infrastructure. You just mix the logits of the reference and teacher at ratio β before distillation. That is the entire implementation.

On GSM8K and MATH this reduced training failure rate from 62% to 7% across 40 independent runs. It also gained 4-6% accuracy on every benchmark. This is a free win. Stop using vanilla OPSD tomorrow.

Reasoning LLMs for SOC triage work, but you have to build the calibrator

Everyone tried throwing LLMs at security alert triage. Everyone got garbage results.

The mistake everyone made was training the model to output a label directly. When you force a model to do chain of thought reasoning first, it becomes dramatically better at classification. But it also completely breaks the label token probability. You can no longer threshold on the output logit to get high confidence triage.

This team did the correct thing: they trained a tiny separate calibrator head that only reads the full reasoning trace and outputs one number: probability the verdict is correct.

The end result: 82.6% overall accuracy. At the operating point used for automated triage, they reduced false positives by 43% and missed threats by 18% over direct label classifiers.

A fine tuned 30B model beat GPT-4o, Claude 3.7 Opus and DeepSeek V4 by 11 points. Scale loses to targeted training here. Always.

We finally have order optimal neural actor critic for constrained MDPs

This is the boring theory paper that will be underpinning every RL alignment method in 12 months.

Until now all primal-dual actor critic methods with neural critics had a hard tradeoff: you could have low bias or low cost, not both. This made convergence guarantees meaningless for production use. Everyone just tuned learning rates until it didn't crash.

This paper introduces hierarchical multilevel monte carlo critics. They run debiasing across both trajectory sampling and critic optimization steps. You get the bias of a 1000 step critic run for log(n) cost.

The resulting algorithm achieves Õ(T^-1/2) optimality gap and constraint violation. This is the theoretical limit. No one will ever get a better bound.

Most importantly: you do not need to know the mixing time of the MDP. This was the single largest unstated requirement that made every prior constrained RL algorithm useless for LLMs.

HARGO: RL post training works for heterogeneous tasks

SFT gives you knowledge. It will not give you good behaviour.

An SFT model that gets 88.6% accuracy on data race detection will also produce 300 word answers to yes/no questions. Everyone has seen this.

RL post training fixes this. Except every standard RL method falls apart when your task set includes binary classification, factual QA and code generation at the same time. GRPO will just optimize for the easiest task and forget everything else.

HARGO solves this with two very simple changes. First it computes group level reward contrast instead of global contrast. Second it modulates advantage using the reference model log probability. No task labels required.

Across four HPC tasks HARGO beat every other method on every metric: 54.6% win rate, 91.3% F1 on data race detection, 0.856 PLP similarity.

This is the first RL post training method that actually works when you have more than one task.

LLMs can rewrite constraint models to make solvers run 10x faster

No one talks about this but 90% of the cost of running combinatorial solvers comes down to how the human wrote the model. Two correct models for the exact same problem can differ by two orders of magnitude in runtime.

This work built an evolutionary search loop where an LLM proposes reformulations, runs the solver, measures runtime, and iterates.

They got average 3.7x speedup across 8 CSPLib problems, maximum 11.2x. The important result was not the speedup itself. Retaining only the fastest previous attempts performed the worst. Retaining behaviourally diverse attempts using MMR on runtime profiles got 2x better speedups.

Optimizers do not care about your loss function. They care about exploring the space you did not think of.

The first good reasoning corpus just dropped

Everyone has been training reasoning models on garbage data until now. Every public dataset either has corrupted traces, no separation between thought and answer, or was generated by a single model with consistent failure modes.

Qyrou released 5 million samples this week. Every sample has separate, normalized fields for prompt, thought trace, final answer, source and precomputed token length.

This is not a pile of scraped chat completions. They merged 89 different upstream sources, filtered, deduplicated and normalized the structure. Most importantly: you do not have to reverse engineer the format. You can train answer only models, explicit reasoning models, distillation targets, or anything else without parsing text.

Use this. Stop scraping ChatGPT.

Method comparison

Method	Fixed failure mode	Implementation overhead	Relative performance gain	Production ready
β-OPSD	Brittle self-distillation	3 lines of code	+4-6%	Yes
CoT + Calibrator	Reasoning breaks confidence	+1 small head	+18-43%	Yes
MLMC Actor Critic	Unstable constrained RL	New critic	+0% / guaranteed convergence	3 months
HARGO	RL fails on mixed tasks	12 lines of GRPO modification	+7-12%	Yes
PDR Evolutionary Search	Bad constraint modelling	Full search loop	+270% runtime speedup	1 month

The unifying pattern across all this work

Every single good result this week follows exactly this pattern. No one is running end to end RL any more. No one is doing pure distillation.

Everyone has stopped trying to optimize the hard objective directly. Everyone is deriving the closed form optimal policy, then distilling that target.

This is the new standard training pipeline. This is what replaced DPO. This is what replaced GRPO. Every independent research group got here independently this month.

If you are still running vanilla DPO or GRPO you are already 6 months behind.

Performance gains by method

Practical takeaways for this week

1. Go change your OPSD implementation to use β=0.7 today. You will not regret it.
2. If you are building any classifier with reasoning LLMs, always add a separate calibrator head. Never use the output logit.
3. Stop trying to make GRPO work across multiple tasks. Port HARGO instead.
4. The reasoning corpus is good enough that you can cancel your next custom data collection run.

None of this work is flashy. None of it got a press release. All of it will be running inside every production LLM by the end of this year.

That is how progress actually happens.