We are not waiting for GPT-7. We are not waiting for new model architectures. Every single production LLM deployed in the next 12 months will run on one or more of the optimizations published in the last 10 days.

None of these works require training a model from scratch. All work on existing frozen checkpoints. All deliver 20-200% improvements on metrics that actually matter for deployment: inference cost, context length, memory usage, and zero measurable drop in downstream quality.

This is not incremental progress. This is the point where everyone's baseline for what a transformer can do just moved.

Stop pruning every layer the same way

Everyone has been doing one-shot pruning wrong. For two years every pruning method from Wanda to SparseGPT has applied exactly the same sparsity ratio to every single transformer layer. Nobody stopped to check if this was a good idea.

PALS shows it was not.

The core observation is brutally simple: layers do not have equal importance. The 99th percentile of activation magnitude for a given layer is an almost perfect predictor of how much weight you can remove from that layer before quality collapses.

PALS does nothing more than adjust per layer sparsity +/-5% around the global target, aligned to that 99th percentile value. That is the entire method. No fine tuning. No extra calibration. 3 lines of code added to an existing pruning pipeline.

This is not a small gain. At identical 50% sparsity, PALS closes 88% of the gap between dense and pruned performance. For reference that is better than most methods that require 1000 steps of fine tuning after pruning.

There is one critical caveat almost nobody is talking about. This effect is architecture specific.

Model	PALS gain over uniform pruning
LLaMA-2 7B	+15.2%
LLaMA-3 8B	+2.1%
Mistral 7B	0.0%

Mistral layers are already well balanced. Meta already fixed this implicitly when they designed LLaMA-3. Everyone else did not.

If you are running LLaMA-2 derived models, you can deploy this next week. Stop reading and go implement it.

RoPE was never broken. You were using it wrong.

Rotary Position Embeddings are the single most copied good idea from the last 4 years of transformer research. Nobody understood how they actually worked until last week.

The paper on RoPE frequency usage kills every single folk myth about RoPE interpolation.

Models do not use all available RoPE frequencies. They ignore 70% of them. The frequencies they do use exactly match the relative distance scales that appear in the training data.

This is not an accident. This is not an emergent quirk. This is optimal behaviour. For a dependency that occurs on average every W tokens, the optimal frequency is exactly 1/W.

Natural language has dependencies at 6, 10, 27 and 49 tokens. That is exactly the four frequencies every trained LLM converges to using.

This explains every success and failure of RoPE scaling. When you scale all frequencies down by 2x to double context length, you are not magically extending position understanding. You are moving every learned frequency band to match twice the distance.

This works only if the dependencies in your longer context are exactly scaled versions of the dependencies seen during training. Natural language happens to have this property. Code does not. Retrieval data does not. This is why everyone reports RoPE scaling works great for chat and falls apart for RAG.

You do not need to fine tune for long context. You just need to leave the high frequency bands alone. Only scale the low frequencies. Every production implementation currently gets this backwards.

Linear attention was missing three trivial fixes

We have known linear attention is possible for 5 years. We also knew every implementation was 5-10% worse than softmax attention. Nobody could explain where the gap came from.

The Key to Going Linear finally isolates the error.

Softmax attention is not magic. It is just a rank 1 orthogonal projection applied at every update step. All prior linear attention implementations replaced this with a gated accumulation. That was the mistake.

Delta style updates preserve this projection property. That is why DeltaNets were always the best linear attention variants, and nobody could explain why.

Once you know this, the remaining gap can be closed with three tiny structural changes:

1. One single sink token per head
2. 3 token depthwise convolution on the value stream
3. Fixed budget cache routing for the oldest 10% of state

That is all.

This works on frozen models. You can take an existing LLaMA 32B checkpoint, replace the attention layers with this implementation, and get linear scaling context up to 128k tokens while matching original MMLU scores. This is not a claim. This is the published result.

Every single long context hack deployed today will be obsolete by the end of this quarter.

FourierQK: The result no one saw coming

This is the weird one. This is the paper everyone will be arguing about for the next year.

If you take the query and key projections before attention, run them through an FFT, zero out all but four frequency bands, run inverse FFT, then run standard dot product attention, you get a 79% reduction in validation loss on character level language modelling.

Not 5%. Not 10%. 79%.

This works even if the filter is fixed and random. It works even better if you learn exactly four frequencies. Those four frequencies always converge to exactly 6, 10, 27 and 49 tokens per cycle.

This is the exact same scale structure we saw in the RoPE paper. This is not a coincidence. There is a universal multi scale structure in natural language that no existing attention implementation was exploiting.

There is one catch. This operation is non causal. It leaks information from future tokens. That is why it works. That is also why you cannot just drop this into a causal LLM today.

But this is the single largest empirical improvement to attention mechanism ever published. Everyone is already working on the causal variant. It will arrive.

Sparse Delta Memory breaks the linear attention ceiling

Linear attention had one unbeatable flaw. Past 64k context, recall fell off a cliff. Everyone assumed this was a fundamental limit of fixed size state.

It was not. It was a limit of dense state.

Sparse Delta Memory replaces the single dense hidden state with a large sparse addressed memory bank. You keep exactly the same FLOP count per token, exactly the same parameter count, but you can scale the effective state size from 4k up to 1 million entries.

Under isoFLOP constraints, SDM outperforms standard softmax attention on every long context retrieval benchmark past 32k tokens. At 256k context it is 27% more accurate.

This is the first linear attention variant that does not trade off long context performance for speed. This is the first one that beats softmax.

Gradients lie about pruning importance

One throwaway result from the PALS paper deserves its own section.

The authors tested the obvious principled approach: allocate sparsity per layer using gradient magnitude, exactly as every theoretical pruning paper says you should.

It performed worse than random allocation.

This is not measurement noise. This is a consistent result across 9 independent runs. Gradient magnitude tells you almost nothing about the impact of permanently removing a weight. This breaks every assumption that has guided pruning research for the last decade.

Nobody knows why this is true yet. But it is now an empirical fact. We have been optimizing for the wrong signal this entire time.

The common thread

None of these papers introduce a fancy new architecture. None of them train a 100B parameter model. None of them have pretty demo websites.

All of them win by doing something that almost no one does anymore: they stopped trying to beat benchmarks. They stopped and asked why the existing thing works.

Every single one of these results could have been published 3 years ago. All of them use existing components. All of them were missed because everyone was building on top of black box transformers instead of opening them up.

What you should test this month

Prioritize in this order:

1. PALS pruning for all LLaMA-2 deployments. 1 day work, 2x inference throughput.
2. Partial frequency scaling for RoPE. 3 day work, double usable context length.
3. Linear attention with sink tokens and delta updates. 1 week work, linear context scaling.
4. Hold off on FourierQK until the causal variant lands.

What this means for the field

We have passed the peak of scaling law hype. We are now entering the era of understanding.

For the last 5 years we have been building engines that work, without any idea why they work. We have been adjusting knobs at random, measuring what makes the number go up.

That period is over.

These papers are not incremental improvements. They are the first proper scientific results about how transformers actually operate. Every optimization we build from this point forward will be based on understanding, not guesswork.

The next generation of LLMs will not be 10x larger. They will be 10x more efficient. And they will run on exactly the same core architecture we have been using since 2017.