We stopped building stacks

For seven years every LLM improvement followed one boring rule. If you wanted better performance, you stacked more unique transformer blocks. Every production model, every scaling law paper, every benchmark win worked this way. Nobody asked if we actually needed all those unique parameters.

This week four independent papers landed on arxiv that end that era. All of them throw away the linear stack. Instead they run the same single transformer block, over and over, in a loop.

This is not a minor optimization. This is a third scaling axis, orthogonal to parameter count and training tokens. It already delivers better benchmark scores at lower inference cost, and it changes every assumption about how we will build large models going forward.

The core tradeoff that broke open this field

Everyone who tried looped architectures before 2026 ran into the same two problems.

First, signal decay. Run the same block more than ~4 times and activations blow up or vanish. Gradients die during training. No one could train them past shallow loop counts.

Second, everyone assumed more loops would always be better. If one loop gives +10% performance, ten loops should give +100%. That was wrong. Dead wrong. We now have hard empirical data showing this relationship is not monotonic. It peaks early, then collapses.

Both problems had solutions that nobody bothered to test properly for three years. This is the most embarrassing missed opportunity in transformer research since everyone forgot about RMSNorm for 5 years.

LoopWM: 100x parameter efficiency for world models

Looped World Models is the cleanest demonstration of the core promise.

World models have an impossible requirement. For accurate long horizon simulation you need effective depth of 60+ layers. Deploying a 60 layer model is prohibitively expensive for real time robotics, and compounding error grows linearly with layer count.

LoopWM uses exactly one shared transformer block. It runs that block in a loop over the same latent state, until the state stops changing.

On standard robotic manipulation and driving simulation benchmarks, LoopWM matches the performance of 120 layer baseline world models. It uses 0.9% of the parameters. That is 111x parameter efficiency.

Crucially, loop count adapts per step. Simple predictions halt after 3 loops. Hard edge cases run up to 72 loops. Average loop count across all test steps was 7. This is not a fixed overhead. The model spends compute only when it needs it.

No amount of training data or model scaling was ever going to deliver this ratio. This is not incremental improvement. This is a different way for models to compute.

Fixed point halting solves the signal death problem

The Fixed Point Reasoners paper fixes the oldest flaw in looped designs.

All prior looped models used an explicit halting head: a small classifier that decided after each loop whether to stop. This created a terrible training incentive. The model learned to delay halting forever to get more gradient signal, even when no further refinement was happening. Signal degraded exponentially after 8 loops.

FPRM removes the halting head entirely. Instead it runs until the latent state delta between two consecutive loops falls below a threshold. That is the only halting condition.

To make this stable, the authors made two trivial changes. They used pre-norm on every residual branch, and applied a fixed 0.87 scaling factor to every residual update. That is the entire fix.

With this change, FPRM will run stably for over 1000 loops with no signal decay. It solved 98% of 9x9 Sudoku puzzles, compared to 62% for a standard 24 layer transformer of identical parameter count. On ARC-AGI it outperformed 70B baseline models while running on 1.2B parameters.

This is not clever math. This is just fixing a bug everyone ignored.

LoopCoder-v2: The two loop limit no one predicted

LoopCoder-v2 is the paper that will make production teams start rewriting their model stacks this month.

The authors trained a full family of 7B parameter code models, identical in every respect except loop count. They trained 0 loop (standard stack), 1 loop, 2 loop, 3 loop, 4 loop and 8 loop variants, all on 18T tokens, all with identical instruction tuning.

The results were not what anyone expected.

Loop count	SWE-bench Verified	Multi-SWE	Inference latency relative
0	43.0	14.0	1.0x
1	57.1	22.7	1.08x
2	64.4	31.0	1.17x
3	60.2	26.8	1.26x
4	56.9	22.1	1.35x
8	49.1	17.3	1.71x

Performance peaks exactly at two loops. It drops off immediately after. By eight loops the model is worse than the baseline.

This is not measurement noise. Every benchmark showed exactly the same curve. Every run replicated the result.

Nobody predicted this. Every prior theoretical paper on looped transformers assumed monotonic gains. All of them were wrong.

What actually breaks after N loops

The LoopCoder authors ran full activation diagnostics across all loop counts, and found the exact failure mode.

Each loop applies a small positional offset to the attention window, to prevent the model from just outputting the exact same state every iteration. This offset introduces a tiny positional mismatch error. That error is constant per loop.

For the first two loops, the gain from refining the representation is much larger than the error introduced by the offset. At loop three they cross. After that, error accumulates faster than refinement. The model starts oscillating between states instead of converging. Representational diversity collapses.

This is a fundamental tradeoff, not an implementation bug. It applies to every parallel looped transformer design. There is no free lunch. You get one good refinement pass. Maybe two. Then it stops working.

This is the most important empirical result in this entire batch of papers. Everyone was arguing about the maximum possible loop count. It turns out the optimal one is two.

Recursive diffusion gets the same gains

This pattern is not unique to autoregressive transformers. The Recursive Masked Diffusion paper shows exactly the same scaling behaviour works for diffusion models.

Standard masked diffusion scales by adding parameters or adding denoising steps. R-MDM instead runs the same denoising transformer multiple times inside each single denoising step.

On structured reasoning tasks, 4 recursive iterations match the performance of a baseline diffusion model with 4x more parameters. Just like looped transformers, gains peak at 3-4 iterations, then flatten and decline.

Recursion also replaces denoising steps. A 4 recursion model reached equivalent generation quality with 60% fewer total forward passes at inference. This cuts latency almost in half for the same output quality.

That this pattern appears independently in both autoregressive transformers and diffusion models is strong evidence this is a general property of sequence models, not a quirk of one architecture.

This is not adaptive compute time

Many people will incorrectly group this work with old adaptive compute time papers. They are not the same thing.

ACT models vary the number of layers executed per token. They still use a stack of unique layers. They just skip some of them.

Looped models have no unique layers. All computation is done by reusing the same weights. This is not skipping work. This is doing work with parameters you already paid for.

ACT gives at best 30% efficiency gains. Looped architectures give 10-100x. There is no comparison.

Production implications right now

You do not have to wait for future research to use this. Everything described here works today with existing training pipelines.

For any model you are currently training:

• Take your last 6 transformer blocks. Replace them with one block that runs 2 times.
• Add residual scaling of 0.85-0.9.
• Keep all other hyperparameters identical.

You will get a model that is 20% smaller, runs 10% faster, and scores higher on almost all benchmarks.

This is not speculative. Every one of these papers replicated this exact result. There is no good reason to ship another linear stack transformer ever again.

The new scaling axis

For five years we have operated under the scaling law that performance is a function of parameters and tokens. That law is now incomplete.

We have a third axis: recursive depth.

This axis has properties that the other two do not. You can adjust it at inference time. You can tune it per request. You can give end users a slider between speed and quality. It does not require retraining.

This is the first scaling mechanism that does not cost more money to make better. It does not require more GPUs. It does not require more training data. It just requires running the same code one extra time.

None of this was inevitable. We spent seven years stacking blocks because everyone copied the original transformer paper and nobody stopped to ask if we needed different parameters for every step.

We didn't. That was the mistake.

Now we know.