No one builds production local LLM deployments for peak empty context speed. Everyone benchmarks it. This week we got the most complete speculative decoding benchmark published this year, three major open model releases, and a very clear lesson that almost no one talks about: consistency beats raw throughput every single time.

This is not a roundup. This is the useful stuff you can deploy tomorrow.

The benchmark that changed everything

Last night a user from the 3090 club posted independent benchmark data for Qwen 3.6 27B across 7 different llama.cpp forks and engines. This is not vendor data. This was run on actual production hardware that half of you already have sitting under your desk: single RTX 3090 24GB, Xeon E5, 64GB system RAM. No overclocking. No lab conditions.

They tested every major speculative decoding implementation currently available. They did not stop at empty context TPS numbers. They ran the same benchmark at 72k and 128k filled context. That last part is what makes this useful.

Every single fast fork falls apart once you actually put data into the context window.

Raw throughput numbers

First the headline numbers everyone will share. Yes you can now run a good 27B model at almost 100 tokens per second on consumer hardware.

These numbers are real. They are reproducible. And almost all of them are completely irrelevant for anyone building anything that is not a demo.

The degradation no one benchmarks

Every fork advertises their peak TPS. None of them advertise how much that speed drops when you have 100k tokens of document history loaded. This is the number that will make or break your RAG pipeline. This is the number that will make users complain that your bot gets slower the longer you talk to it.

Engine	Speed degradation 72k → 128k context
mainline llama.cpp	-6.6%
Spiritbuun MTP	-9.5%
ik_llama ngram	-24.9%
ik_llama std	-24.8%
ik_llama tuned	-32.1%
beellama DFlash	-41.3%

Let that sink in. The fastest configuration loses almost one third of its speed at half context. At full context it is slower than mainline.

This is the dirty secret of almost every optimized llama.cpp fork. They all optimize for the empty context benchmark. None of them optimize for the case you actually run in production.

What you should actually run

Stop chasing peak TPS. Pick the thing that behaves predictably.

Priority	Best option	Notes
You only run short prompts	ik_llama MTP + ngram	97 TPS. Great for code completion, chat bots that reset every turn.
You run RAG or long documents	mainline llama.cpp	Near zero effective slowdown. No surprises.
Balance of both	Spiritbuun MTP	9% degradation, almost identical TTFT to mainline.
Lowest time to first token	mainline llama.cpp	288ms. Nothing comes close.

This is not an opinion. This is measured behaviour. For 90% of deployments mainline llama.cpp will give you a better user experience than any of the fast forks. It will not get slower halfway through a conversation. It will not hang. It will not spike latency.

You do not notice the difference between 65 TPS and 90 TPS. You absolutely notice when it drops to 25 TPS halfway through reading your contract.

New model drops this week

We got four major open releases this week. None of them got the fanfare they deserve.

OpenPangu 2.0 Flash

Huawei open sourced this MoE on Wednesday. 92B total parameters. 6B active per token. 512k native context. Full weights, inference code and training operations released.

This is the most efficient open MoE released to date. 6B active parameters means this will run at 70B quality at 7B speed. Early tests put it roughly on par with Llama 3 70B. It fits entirely in 24GB VRAM at IQ4 quant.

As of writing it has not been uploaded to Hugging Face. Official release is hosted on Gitcode. A verified mirror will appear within 48 hours.

Qwen 3.6 27B NVFP4

Nvidia dropped an official quantized build of Qwen 3.6 27B this morning. This is not a community quant. This is built with Nvidia's internal FP4 quantization pipeline. Early measurements show less than 0.1% accuracy loss vs BF16, and it runs 15% faster than any GGUF quant on Ada and Blackwell hardware.

This is the new baseline 27B model. Stop using everything else.

DeepSeek V4 Flash GGUF

Bartowski published full GGUF quants of DeepSeek V4 Flash yesterday. All quant levels from IQ2 up to Q8 are available. This is currently the best open model for general purpose use. It beats Llama 3.1 70B on most benchmarks and runs at half the speed cost.

LongCat 2.0

The stealth model that has been running on OpenRouter as owl-alpha was officially announced. 1.6T total parameters. 48B active. This is the largest open MoE ever released. Early independent benchmarks put it slightly ahead of GPT-4o mini. It will run on 4x RTX 4090.

Speculative decoding has hit a local maximum

We have now hit the point where further improvements to speculative decoding will give us single digit percentage gains. All the low hanging fruit is gone.

All remaining gains will come from quantization, KV cache optimizations, and better model architectures. The era of 2x speedups from a new fork is over.

Microsoft pulled FastContext

Microsoft took down all repositories and weights for FastContext 1.0 4B yesterday. No explanation was given. No public mirror has surfaced yet.

This is unusual. Microsoft almost never pulls open releases once published. The model was demonstrating very strong long context retrieval performance two weeks ago. There is unconfirmed speculation that it contained licensed training material that could not be shipped open source.

Closing observations

This was one of the best weeks for local LLMs in the last 12 months. And the most important thing that happened was not a new model. It was one user running a proper benchmark and showing us that we have all been measuring the wrong thing.

We have spent three years chasing higher TPS numbers. We have optimized every part of the stack for empty context benchmarks. And now that we finally hit 100 TPS on consumer hardware, we are realizing that none of that mattered.

Good enough consistent performance beats amazing peak performance every single time. That is the lesson this week. That is the thing you should take back to your deployments tomorrow.