Right now, almost every large language model on the planet is trained with AdamW. Every reference implementation, every cloud training template, every internal runbook defaults to it. And we just found out we have no proof it actually works under the conditions it is always used in. That is not an exaggeration. That is the headline finding from four optimizer and training dynamics papers that dropped on arXiv the same day last week.

This is not incremental work. This batch of research breaks assumptions that have been baked into production ML pipelines for seven years. None of it requires new hardware. None of it requires changing model architectures. All of it directly affects the cost, stability and final quality of every model you will train this year.

Nobody proved AdamW works with heavy tailed noise

All existing convergence proofs for AdamW operate under one non-negotiable assumption: stochastic gradient noise has finite variance. For fifty years this was the default assumption for every optimizer paper. Nobody bothered to check if it was true for real LLM training.

We now know it is false. Empirical measurements across every model size and architecture show LLM pretraining gradients follow alpha-stable distributions with tail exponents consistently around 1.8. For any alpha < 2, variance does not exist. The entire mathematical foundation that AdamW was built on does not apply.

Lion, Muon and AdaGrad all have published convergence proofs that hold under heavy tailed noise. AdamW does not. Worse, the authors of the open problem paper demonstrate an explicit failure mode unique to AdamW: the exponential moving average second moment accumulator will smooth over large rare gradients until they are effectively erased from the update. The optimizer will stop making progress, and you will never see any warning in your training logs.

This is not an edge case. This is every LLM training run you have ever run.

The open problem every training engineer should care about

The authors do not claim AdamW fails. They prove we do not know if it works. They produced a partial positive convergence bound under weighted metrics, and also constructed an explicit lower bound corridor where AdamW will provably diverge. Nobody knows which side real production training runs fall on.

Right now we are running the most expensive compute jobs in human history on an optimizer that might be silently failing half the time. There is no workaround. There is no diagnostic. Nobody has even run a controlled test to check.

This is the most important open problem in all of machine learning right now. It is not theoretical. The answer will change how every foundation model is trained.

Muown was not a hack

For the last six months everyone competing in the nanoGPT speedrunning discord has been using Muown. It consistently beat every other optimizer by 15-20%, and nobody could explain why. Most people wrote it off as another heuristic that happened to win on benchmarks, that would break when scaled.

This paper proves exactly what Muown does. It is not a messy combination of two different optimizers. It is a clean geometric algorithm. Muown separates every weight matrix into row magnitudes and unit direction vectors. It runs Adam only on the scalar magnitudes, and runs Muon only on the normalized directions.

The entire observed stability of Muown comes from one property no one had noticed: this parameterization automatically maintains an almost perfectly constant angular step size for the entire duration of training. While global scalar step size is being scheduled up and down, the actual angle that weights rotate each step stays within 2% of its initial value.

Angular step size is the thing nobody was measuring

For ten years we have spent millions of compute hours tuning and scheduling global scalar step size. No one was measuring or scheduling the angular change per parameter step. That is the actual quantity that controls training stability.

When people said Muown "never blows up" this is why. Standard optimizers will silently increase angular step size by an order of magnitude as parameter magnitudes grow during training. Eventually it crosses the stability threshold and the run diverges. Everyone blamed bad learning rate schedules. It was never the schedule. It was the parameterization.

AngularMuown benchmark results

Once the authors understood this mechanism, they built AngularMuown: an optimizer that makes the angular step size an explicit, schedulable parameter completely decoupled from magnitude updates. It is not a minor tweak.

Optimizer	nanoGPT 124M steps to 3.0 val loss	Qwen2-0.5B steps to 2.7 val loss	Relative wall time
AdamW	13800	21100	1.00x
Muon	11200	17900	0.97x
Muown	10100	16300	1.01x
AngularMuown	8900	14700	1.02x

This is a 22% reduction in total training steps for exactly the same final loss. On a 16 day run across 1024 A100s that is three and a half days saved. No architecture changes. No data changes. No tuning. Just using the correct optimizer.

As of this week AngularMuown holds first place in every category of the nanoGPT speedrun leaderboard. Results have also been replicated on 1.1B MoE models. There are no reported scaling failures so far.

GD convergence finally leaves the NTK regime

For seven years every convergence proof for neural networks lived in the Neural Tangent Kernel regime. That is the regime where the network is effectively linear, weights barely move during training, and the model behaves nothing like actual trained transformers. All theory was being done for a toy model that no one actually uses.

This paper is the first general convergence proof for plain gradient descent that applies to real pre-norm transformers, at standard Xavier initialization, with practical learning rates. It does not require infinite width. It does not require infinitesimal step sizes.

The proof also produces an explicit scaling law for maximum safe learning rate: it scales with the inverse square root of network depth, not inverse width. This matches every empirical observation from the last three years that no one could explain.

We are no longer doing theory that runs parallel to practice. We now have a framework that can actually predict the behaviour of training runs you will run tomorrow.

SOAP gets proper uncertainty for free

Proper weight uncertainty estimates have been one of the most requested features in deep learning for a decade. They enable good OOD detection, calibrated probabilities, smart pruning and transfer learning. Until last week every usable implementation added 300-400% overhead. No one ran them at scale.

EVON changes this. The algorithm reuses the eigendecomposition that the SOAP optimizer already computes every training step. Instead of throwing away the eigenspace after preconditioning, it tracks a simple diagonal variance estimate there. When projected back to weight space this gives a full rank covariance estimate for every weight matrix for ~2% overhead over base SOAP.

You get all the benefits of Bayesian neural networks for effectively free. No changes to your training loop. No extra hyperparameters. Just swap the optimizer.

Optimizer execution flow comparison

What this means for your training runs

You do not need to rewrite your entire pipeline next week. But you should stop defaulting to AdamW for every new run. Run one side by side test with AngularMuown on your next small model. Add a log line that tracks tail exponent of gradient noise. Stop treating optimizers as a solved problem.

The difference between a good optimizer and a bad one is 20-30% of your entire training budget. That is larger than every other optimization you will do combined. Larger than kernel fusion. Larger than distributed training improvements. Larger than any hardware generation upgrade.

Closing observation

None of this work came from OpenAI, Google Deepmind or Meta. All four papers came from independent researchers and small university groups. All of them are explaining effects that people have observed empirically for years, but nobody bothered to prove.

There is no secret sauce in large model training. There is just basic work that nobody has done yet. We are still in the very early days of understanding how these systems actually learn. The most impactful improvements will not come from bigger models. They will come from understanding the algorithms we have been running this entire time.