The quiet inference revolution no one is announcing

If you only read press releases you would think all progress in AI is still about bigger models, higher benchmark scores, and new cloud API endpoints. That is not what is actually happening.

Every single important paper, release and field report from the last two weeks is about exactly the opposite. It is about running less of the model. Running it faster. Running it on hardware you already own. Fixing the stupid trivial bugs that were wasting 90% of your compute. And doing all of this without retraining, without more pre-training compute, and without asking permission.

This is not incremental improvement. This is a phase shift. The era of scaling models is over. The era of making inference not suck has just started.

MoE has won, and no one told you

Two separate MoE models dropped this month that completely redefine the baseline.

Soofi S 30B-A3B activates only 3B parameters per token. It matches or beats dense 14-27B models on every benchmark. It has near constant cache size with context length. It was trained on sovereign European hardware, and will be released fully open.

Mach-Mind-4-Flash is also 3B active parameters. It beats 100B class models on agent and reasoning benchmarks. It did this without scaling pre-training. All gains came from post training optimization and distillation.

Model	Total parameters	Active per token	AIME 2026 score	Inference cost relative to Llama 3 70B
Llama 3.1 70B	70B	70B	71.2	1.00
Mach-Mind-4-Flash	35B	3B	92.7	0.07
Soofi S 30B	30B	3B	N/A	0.06

This is not a small difference. This is a 14x improvement in cost per quality. No dense model will ever catch this. Every new foundation model released from this point on will be a sparse MoE. There is no reason to build anything else.

Test time training stops lying about long context

Everyone knows long context is broken. You can extend a model's window to 1 million tokens. It will still not find the one sentence you asked for buried on page 72.

Self-Guided Test Time Training fixes this properly. Instead of trying to train the model to remember everything forever, you do 30 steps of fine tuning on exactly the relevant parts of the context at inference time, right before you ask the question.

This works. It delivers up to 15% relative improvement on LongBench-v2. It works on every existing model. It requires no retraining. It adds almost no overhead for long documents, because you only run adaptation on the 1-2% of spans the model already thinks are relevant.

No one will be serving vanilla long context models 12 months from now. This will be standard in every inference engine.

NPU inference is not what vendors sell you

Every laptop and phone ships with an NPU now. Every vendor will tell you it runs LLMs fast. None of them will tell you that all of the reference implementations are at least 10x slower than they should be.

STEEL is an open source FlashAttention implementation for AMD XDNA. It gets 9.6x lower latency and 9.17x lower energy use than the official AMD implementation. It does this by not doing the stupid things the vendor stack does. It understands sparsity. It schedules work properly. It does not move data around for no reason.

The Gemma 4 Inferentia port tells exactly the same story. The official AWS stack could not run the model at all. It produced gibberish. By bypassing every single vendor abstraction layer and tracing the raw forward pass directly, the port got correct, fast inference running at 44 tok/s on the smallest Inf2 instance.

Vendor inference stacks are not optimized for performance. They are optimized for vendor lock in. You will always beat them by ignoring 90% of what they give you.

The 90% gain you get from not breaking the cache

The single largest performance improvement reported this month did not come from a new algorithm, a new model or new hardware. It came from fixing three bugs.

A developer running Qwen3.5-122B on a Mac Studio had prefill times of 3-5 minutes per turn. After fixing:

1. A unique message id that broke KV cache matching
2. Interrupted generations that poisoned cache history
3. A background writer that created invalid cache checkpoints

Prefill time dropped to sub-second. For 168,000 token contexts, time to first token went from 5 minutes to 2.6 seconds.

This is the dirty secret of production LLM inference. 90% of your performance loss is not architecture. It is bugs. It is bad defaults. It is unnecessary work that no one ever bothered to remove.

Super-tuning: pruning was always fine tuning

Super tuning is the simplest good idea that no one thought of for three years.

All of the saliency metrics that people invented for pruning work perfectly for selecting which parameters you should fine tune. You run one calibration pass. You pick the 1% of parameters that actually activate for your task. You only fine tune those.

You get equal or better accuracy than LoRA. You get no adapter overhead at inference time. You use 100x less memory during fine tuning.

This changes everything for fine tuning. There is no reason to use vanilla LoRA for most tasks any more.

Heretic: the most important LLM tool released this month

Heretic removes alignment from models automatically. It does this in 20 minutes on a consumer GPU. It produces models that have the same refusal rate as manual abliterations, but with 1/3 the KL divergence from the original model.

Method	Refusals / 100	KL divergence
Original Gemma 3 12B	97	0
Manual abliterated v2	3	1.04
Manual abliterated best	3	0.45
Heretic automatic	3	0.16

This is not just a tool for removing censorship. This is the first general purpose tool that can reliably remove a specific learned behaviour from a trained transformer, without retraining, and without breaking everything else. This is interpretability delivering actual usable results.

Everyone working with fine tuned models will be using this or something very much like it by the end of the year.

Inference configuration is not free

A large controlled study of vLLM configurations measured 9000 separate runs across 5 models and 5 tasks. The conclusion was unambiguous: inference engine settings change energy use by up to 7x, latency by up to 11x, and even change output accuracy.

There is no universal optimal configuration. The best settings for code generation are terrible for summarization. The best settings for a 7B model will destroy performance on a 70B model.

Almost no production deployments tune these values today. Most teams just run the default config and accept that they are throwing away 2/3 of their hardware capacity.

The end of the cloud monopoly

PrismML got a full 27B dense model running on an iPhone. Not 27B total with 3B active. All 27B parameters active. It fits in 4GB. They claim no meaningful accuracy loss.

Someone got Gemma 4 running inside Godot using only GDScript and Vulkan compute shaders. No llama.cpp. No Python. No server.

122B models run on consumer workstations. 30B models run on laptop NPUs. 27B models run on phones.

Cloud providers still sell you inference at $0.50 per million tokens. That price will collapse by 90% in the next 18 months. There is no way around it. The economics no longer work.

What comes next

None of this work was done by OpenAI, Anthropic, Google or Amazon. All of it was done by independent researchers, small teams, and random people on the internet. All of it was released open source.

The centre of gravity has already shifted. The big labs are still fighting over who has the biggest model. Everyone else already moved on to the actual hard problem: making this technology work, everywhere, for everyone, without asking permission.

We are not waiting for the next big model release. We are building the world that comes after.