Every single LLM research paper dropped on arxiv this week says the same thing, if you read between the lines. We are not getting meaningful gains anymore from making models bigger.

All of the progress right now is going backwards. We are fixing the mistakes we made when we just threw compute at every problem for four years.

This is not an opinion. This is the consistent conclusion across six independent research teams working on every core part of LLM engineering. This article walks through what they found, what works, and what this means for anyone deploying LLMs in production.

We stopped winning with scale

For half a decade the entire field operated on one simple rule: double compute, get predictable linear improvements in loss. That rule is broken.

Nobody announced this. Nobody wrote a press release. But every working group publishing this month is now operating from the unstated assumption that training larger base models is no longer the highest return investment. All six papers covered here address failure modes that scaling was supposed to solve. None of them propose larger models as the solution.

This is a quiet turning point. You will not see it in vendor announcements for another 12 months. But it has already happened.

The scaling exponent dead end

Scaling exponents are not abstract numbers. They tell you how much additional performance you get for every extra dollar you spend on training.

The paper On the Smallness of the Large Language Models Scaling Exponents quantifies what everyone already suspected: current exponents are now so low that doubling compute yields less than 3% improvement in downstream loss. At these values, the energy required to move the needle even slightly exceeds the practical value of the improvement for every commercial use case.

Critically, the authors show that the widely cited "pedestal effect" correction does not fix this. Even after accounting for asymptotic minimum loss, the scaling trajectory remains unsustainable. We are not just approaching the wall. We have hit it.

Plasticity loss does not go away when you get bigger

Plasticity loss is the ability of a model to learn new information after it has already been trained. Until this month, almost everyone assumed this was only a problem for small models. It was widely believed that larger transformers simply did not suffer this limitation.

That belief was wrong.

Researchers tested GPT style models from 5M up to 314M non embedding parameters on a controlled multilingual continual learning task. Plasticity loss was observed across every single model size. Larger models delay the onset of measurable loss. They do not eliminate it.

The effect scales sublinearly. Extrapolated out, even a 1 trillion parameter model will lose most of its ability to adapt after roughly one million training steps. This is not a bug that can be fixed with more layers. This is a fundamental property of the transformer architecture as currently implemented.

Worse, the effect appears even during stationary training on a single mixed dataset. You do not need hard task boundaries to trigger plasticity loss. It happens naturally just from training for long enough.

Distillation scaling laws are not what you thought

Until this paper, distillation was treated as a messy empirical process with no predictable rules. Teams would run 20 different experiments and pick the one that worked.

Scaling Laws for Task-Specific LLM Distillation changes this. The authors derived consistent, repeatable scaling relationships for compression performance across four axes: dataset size, compression ratio, supervision format and pruning schedule.

The single most important finding: general knowledge collapses 2-3x faster than in-domain task performance during compression. Everyone knew there was a tradeoff. Nobody knew how large, or how predictable it was.

Chain of thought supervision almost completely eliminates this gap. For the same compression ratio, models distilled on reasoning traces retain 92% of general knowledge performance compared to 41% for models distilled only on final logits. This is a free improvement. It requires no extra parameters, no extra compute at inference. Almost no production teams were doing this correctly before.

Cross architecture distillation finally works

Heterogeneous distillation, transferring knowledge from one architecture type to another, has been broken for years. You could distill a transformer into a smaller transformer. You could not reliably distill a transformer into a CNN, RNN or any other faster architecture without massive quality loss.

Heterogeneous Knowledge Distillation via Geometry Decoupling explains why. All prior work missed two coupled failure modes:

1. Massive feature norm discrepancies between architectures that create optimization drag
2. Severe gradient conflicts between task loss and distillation loss

The proposed SPOFA framework fixes both with less than 0.1% additional parameters on the student model. It explicitly decouples feature magnitude from direction before alignment, and uses a momentum gradient scaler to suppress conflicting distillation signals during training.

In benchmarks, SPOFA achieves 98% of teacher model performance when distilling a 7B transformer into a 1.2B CNN. That CNN runs 12x faster on commodity hardware than the original transformer. This was not possible 12 months ago.

KV cache was the bottleneck no one admitted

Long context inference costs are dominated almost entirely by KV cache memory and bandwidth. For 128k context sequences, over 95% of inference runtime is spent moving KV cache entries between memory levels.

Every existing eviction method was doing the same stupid thing: averaging attention scores across every attention head to decide which tokens to keep.

CompressKV demonstrates that 97% of attention heads contribute nothing to semantic retrieval. Only 3% of heads actually store and recall meaningful information from context. All other heads can be completely ignored when making eviction decisions.

Method	KV Cache Retention	LongBench QA Accuracy	Needle Haystack Accuracy
Full Cache	100%	89.2%	99.8%
H2O	10%	71.4%	62.1%
StreamingLLM	10%	76.8%	78.3%
CompressKV	3%	86.7%	97.2%
CompressKV	0.7%	72.1%	90.1%

This result should embarrass every LLM vendor that has been charging a 10x premium for long context. They have been wasting 97% of the memory you are paying for. CompressKV is open source, works on all existing GQA models, and requires no retraining. You can deploy it this week.

Non autoregressive generation is actually usable now

Non autoregressive generation has been five years away for five years. Every prior implementation either produced garbage output, or required so many refinement steps that it ended up slower than standard decoding.

Posterior Refinement: Fast Language Generation via Any-Order Flow Maps is the first implementation that breaks this tradeoff. FMLM+ produces full sequence output in one pass, then runs a global consistency check to identify low confidence tokens. Only those tokens are regenerated.

This approach matches autoregressive output quality with 32x fewer function evaluations. Unlike all prior non autoregressive methods, it preserves the ability to edit arbitrary subsets of the sequence after generation. You can go back and correct a single wrong number in a 1000 token response without regenerating anything else.

What this means for production deployments

Stop waiting for the next big model. Every single improvement described here can be applied to existing models that you can run today. You can get an order of magnitude better cost performance right now, without waiting for any vendor release.

This is not incremental. This is a step change. Teams that adopt these methods over the next six months will be operating on an entirely different cost curve than teams that continue to run stock base models.

You will not be outcompeted by someone running a larger model. You will be outcompeted by someone running the same model as you, but 10x cheaper and 5x faster.

The coming end of raw scaling

For four years every major LLM announcement was just a bigger model. That era is over.

The entire field has pivoted. None of the top working groups are publishing work on scaling larger base models anymore. All of the good work is now about fixing the broken parts of the systems we already built.

This is good news. We spent half a decade learning how to build very large, very inefficient models. We are finally learning how to build good ones.