Three papers dropped on arXiv within 48 hours last week, all attacking the exact same broken problem, all arriving at working independent solutions. This is not a coincidence. This field just converged.

For nine months on-policy self-distillation (OPSD) was the worst kept open secret in LLM training. Everyone used it. Everyone knew it destroyed general reasoning. No one would admit it publicly. All three papers lead with the exact same damning baseline numbers that every lab had been hiding.

The broken promise of self distillation

OPSD looked perfect on paper. You take one model. Run it once with the reference answer visible, call this the teacher. Run it again blind, call this the student. Distill teacher logits onto the student trajectory. No human labels. No external data. Free performance.

It did not work. It never worked. Every single team that ran this got the exact same result: 8-12% improvement on the training benchmark, 10-15% collapse on all out of distribution tasks. The model turned into a benchmark parrot. It forgot how to think.

What actually breaks in standard OPSD

This is the standard OPSD training loop used by every lab through June 2026:

All three papers independently confirmed exactly three failure modes in this loop:

1. Privileged information leakage. The teacher sees the answer. It produces logits that only make sense if you already know the answer. The student learns to mimic those logits, not reason. At test time there is no answer. The student has nothing.
2. Exploration collapse. Dense token level distillation crushes output entropy. The model stops trying anything new. It will never discover a better reasoning path than the one the teacher already took.
3. Reasoning erasure. For models that already produce valid long chain of thought, OPSD actively deletes that capability. It replaces general reasoning with rote pattern matching to reference trajectories.

Baseline failure benchmarks

This table is replicated identically across all three papers. Every method that existed before last week traded general reasoning for benchmark score.

Method	SciKnowEval In-Domain	GPQA Out-of-Domain	Training Entropy	Calibration Error
Base Llama 3 70B	62.1	51.3	2.72	11.2%
Standard OPSD	71.4	44.7	1.18	26.8%
GRPO	68.9	48.2	1.91	19.4%
SDPO	67.7	47.1	1.76	21.1%

This was the great unspoken tradeoff of LLM post training. You could make the number go up, or you could keep a model that actually works. You could not have both.

DemoPSD: Disagreement modulated blending

DemoPSD fixes this at the loss function. It does not distill the teacher. It distills a reverse-KL barycenter: a weighted geometric blend between teacher and student distributions.

At every single token position, it measures divergence between the two distributions. If they agree strongly, blend mostly teacher. If they disagree strongly, blend mostly student.

That is the entire trick. It is 7 lines of code changed in the loss calculation.

The authors prove formally this construction both attenuates privileged leakage and preserves exploration capacity. No new hyperparameters. No extra forward passes.

DemoPSD scores 70.2 on SciKnowEval, 52.4 on GPQA. It beats every baseline in domain and outperforms the original base model out of domain. Training entropy stays at 2.47 through full training. It did not kill the model's ability to think.

Neuron-OPSD: Stop training on garbage trajectories

Neuron-OPSD attacks the problem from the data side, not the loss side.

Everyone was running 8 rollouts per question, taking majority vote, training on all of them. That is stupid. 70% of generated trajectories are garbage. Most of the time the model itself already knows it produced garbage.

Neuron-OPSD reads internal neuron activations. It identifies which rollouts actually activated the model's general reasoning circuits, and which only activated shortcut pattern matching circuits. It discards every trajectory that did not trigger real reasoning, before any distillation runs.

No change to loss function. No change to distillation target. Just stop training on the garbage that the model itself already knows is wrong.

This method gets 69.8 in domain, 51.9 out of domain. Calibration error actually drops to 9.7%. Every other post training method ever invented makes calibration worse. This one makes it better.

Purified OPSD: Subtract the answer shortcut

Purified OPSD solves the problem for long chain of thought models, the class of model that standard OPSD breaks most catastrophically.

The authors ran one additional control forward pass. Same model, given only the answer, no question. That gives you the exact logit distribution the model produces purely from knowing the final answer.

They subtract this distribution from the teacher distribution. The residual is the actual reasoning signal. That residual is what you distill.

You literally subtract the cheat code.

On long CoT MATH 500, standard OPSD gives +2% accuracy while destroying reasoning quality. Purified OPSD gives +11% accuracy. The model continues to produce original, valid 20+ step reasoning chains after training.

Side by side method comparison

None of these approaches are competing. They fix orthogonal failure modes. None of them conflict. You can and should stack all three.

Approach	Core mechanism	Fixes leakage	Preserves exploration	Preserves long CoT	No extra hyperparameters
Standard OPSD	Full teacher distillation	❌	❌	❌	✅
DemoPSD	Reverse KL barycenter blending	✅	✅	❌	✅
Neuron-OPSD	Activation filtered rollouts	✅	✅	✅	❌
Purified OPSD	Reference residual subtraction	✅	❌	✅	✅

All three are fully compatible. Early unofficial testing posted over the weekend shows stacking all three delivers an additional 4-6% across all benchmarks with no observed generalization decay.

Iterative training stability

This is the result that changes everything. All three papers ran 8 consecutive full distillation cycles.

Standard OPSD falls off a cliff. Every cycle makes the model dumber. DemoPSD just keeps going up, slowly, reliably, cycle after cycle. There is no observed ceiling at 8 cycles. The authors stopped testing because they ran out of compute.

This is the first training loop ever demonstrated that does not rot the model while it improves it.

Open questions

Nobody has tested this above 70B parameters. All experiments ran on Llama 3 70B. We do not know if this behaviour holds at 400B or 1T scale.

Nobody has run more than 8 cycles. We do not know if improvement continues indefinitely, or if it asymptotes at some fixed capability level.

Nobody has properly measured what else changes during this training. The model gets better at benchmarks. We do not know what else it gets better at. We do not know what it loses.

Implementation notes

All three methods are less than 50 line changes to a standard OPSD training loop. None require extra model parameters. None require external data. None require human annotation.

You can implement all of this this week on your existing training stack.

DemoPSD is 7 lines modifying the target distribution before loss calculation. Neuron-OPSD is a rollout filter with an activation threshold. Purified OPSD is one extra forward pass per batch.

There is no excuse to still be running standard OPSD.

This is not an incremental improvement. This is the end of the first era of LLM post training. For the first time we have a stable loop for iterative self improvement. We do not know where it stops.