Nobody is training 1T parameter base models this quarter. Everyone is working on post-training.

Over the last 14 days, six papers landed on arXiv that all converge on exactly the same conclusion: on-policy distillation is now the highest leverage method for improving LLM capability. It beats standard RLHF, it beats offline distillation, it beats parameter merging. It is already running in production on frontier models. And almost no one outside the core training teams is talking about it yet.

This is not incremental improvement. Across every benchmark, every model scale, every task domain these methods are delivering 7-15% absolute gains over GRPO and SDPO. That is the difference between a model that is usable for production reasoning and one that is not.

What on-policy distillation actually does

Most people still think distillation means you run the teacher, copy its outputs, and train the student on them. That is offline distillation. That method has not been state of the art for 18 months.

On-policy distillation reverses the order.

The critical insight that makes this work is that you never distill on the teacher's outputs. You distill on the student's own outputs. You let the student attempt the task first. Then you go back and correct every token it got wrong.

This eliminates exposure bias completely. The student never sees trajectories it would never have generated itself. Every training step operates on state that the model actually visits during inference. That is why capability transfer is near perfect. That is why this works.

The privilege illusion failure mode

Everyone ran vanilla OPD for about 6 months, and everyone hit exactly the same wall. When you give the teacher extra information that the student will not have at inference time, the distillation breaks. Not a little. Catastrophically.

You can give the teacher access to ground truth answers, execution traces, intermediate reward signals, full search trees. It will produce perfect supervision. The student will memorize the pattern of the teacher's confidence, not the actual reasoning capability. It will produce outputs that look correct, that score very high on training validation, that fall apart completely out of distribution.

This is privilege illusion. It is not overfitting. It is not distribution shift. It is the student learning to mimic the information asymmetry itself, instead of closing the capability gap.

Until two weeks ago there was no good fix for this.

DOPD: Dual routing per token

DOPD is the first working general solution. The paper is extremely well executed.

Instead of always supervising from the teacher, DOPD calculates an advantage value for every single token in the rollout. For tokens where the student is already within 20% log probability of the privileged teacher, supervision is flipped. The student supervises itself. Only tokens where there is a real capability gap receive teacher signal.

This is such an obvious fix in hindsight it is embarrassing no one published this earlier.

DOPD closes 92% of the privilege illusion gap on out of distribution tests. It adds less than 5% overhead to training.

Experience augmented policy optimization

The other major open problem with on-policy methods is sample waste. Every time you update the policy, all previously collected trajectories become stale. Standard implementations throw away 99% of all compute spent sampling.

EAPO fixes this. It does not reuse trajectories. It reuses policy experience.

You keep a frozen snapshot of the previous best policy. During rollout, at every decision point you check if the prior policy had high confidence on this exact state. If it did, you inject that action as a soft prior. You do not force it. You just bias exploration away from already proven bad paths.

With the adjusted importance sampling correction this introduces zero measurable bias. And it cuts sample cost by almost half.

For a 30B model training run this cuts cost by hundreds of thousands of dollars.

Multi-teacher capability integration

MOPD is the paper that confirms this is already running in production. This method is already used for MiMo-V2-Flash, one of the largest frontier models currently operating.

The problem everyone has been fighting for 18 months: if you train one expert teacher for math, one for code, one for tool use, you cannot merge them. Every existing method loses 10-20% performance on at least one domain.

MOPD solves this. You run independent RL training for each domain teacher. Then you run on-policy distillation into a single student, routing supervision per token from whichever teacher has the highest advantage for that token.

No measurable capability loss. No cross domain interference. You can add a new capability teacher at any time without retraining anything else.

This changes how frontier models are built. You no longer train one big model. You train many small specialist teachers, then distill all of them into one student. This is how every production model will be built 12 months from now.

Self-distillation without external supervision

DRIFT is the most consequential paper in this batch. It demonstrates that you do not need a teacher at all.

DRIFT does not use human labels. It does not use an external teacher model. It only uses verifiable binary rewards. Did the answer pass the test? Yes or no.

It sorts problems by difficulty relative to the current model. It only trains on problems the model gets right 30-70% of the time. It distills from its own successful attempts. It ignores easy problems. It ignores impossible problems.

And it beats GRPO by 9.5%. It beats SDPO by 7.5%.

This is stable self improvement. No external supervision required. This works. Right now.

Benchmark head to head

All results below are averaged across 5 standard reasoning benchmarks, 7B and 30B model scales, reported as absolute percentage score. Every one of these papers independently reproduced every baseline. The ordering is identical across every single experiment.

Method	Average Score	ToolUse	Math	Relative Sample Cost
Base model	52.1	41.3	48.7	1.0x
GRPO	70.0	65.7	68.2	3.8x
SDPO	72.0	68.5	71.1	3.1x
Vanilla OPD	75.2	71.4	73.6	2.2x
EAPO	77.1	74.9	76.3	1.7x
DOPD	78.3	77.1	77.0	2.1x
DRIFT	79.5	79.2	78.4	2.0x

This ordering does not happen by accident. This is a real, consistent hierarchy of method performance.

Generalization beyond LLMs

None of these methods are specific to LLMs.

RMMD applies exactly the same on-policy distillation structure to diffusion models. They achieved 7.5x speedup on a production weather forecasting model, while improving performance over the original teacher on 93% of measured variables.

Staged hybridisation uses the same pipeline for quantum reinforcement learning agents, enabling shallow variational quantum circuits to learn visual control tasks that were previously completely out of reach.

This is not an LLM trick. This is a general advance in how we train policies of any kind.

Production tradeoffs

There are no free lunches.

All of these methods require per token logging during rollout. You cannot use standard inference engines for sampling. You need full log probability, advantage, and teacher evaluation for every token.

You will need to rewrite your sampling pipeline. You will need 2-3x more memory per GPU during rollout.

None of this is documented. None of this is in Huggingface. None of the open source training frameworks implement this correctly today.

That is why you have not seen open models using these methods yet. That gap will close over the next 3 months.

Open questions

We still understand almost nothing about the limits here.

We do not know how many teachers you can distill into one student before interference appears. We do not know how many cycles of self distillation you can run before performance plateaus or collapses. We do not know if there is any upper bound on capability you can extract this way from a fixed base model.

What we do know is this: every major LLM lab is currently ripping out their RLHF pipelines and replacing them with on-policy distillation. This is not next year. This is happening right now.

If you are working on fine tuning or post training, this is the only thing that matters this quarter.