Everyone is measuring the wrong thing.

For the last three years every LLM benchmark, release post and forum argument has lead with tg128 — tokens per second generation speed. This number is useless for 90% of actual production workloads people run today.

Over the last two weeks independent engineers ran controlled benchmarks across 13 models, deployed the first open 1.6T parameter MoE, fixed KV cache quantization bugs that have been plaguing everyone for 6 months, and produced hard data that upends almost every conventional wisdom about open weight deployment. This article synthesizes all of it.

The prefill dominance result

This is the single most important number to come out of any benchmark this year. For a typical agentic request: 65,000 tokens of context loaded, 300 tokens of output generated.

Model	Type	Prefill time	Decode time	Total time	Prefill % of total
Trinity-Mini	MoE 3B/26B	46.2s	2.0s	48.2s	96%
Qwen3.6-35B-A3B	MoE	51.7s	2.7s	54.4s	95%
Ornith-9B	Dense	51.4s	3.3s	54.7s	94%
Gemma-4-26B-A4B	MoE	64.0s	2.5s	66.5s	96%
North-Mini-Code	MoE	72.8s	2.2s	75.0s	97%
Granite-4.1-8B	Dense	148.4s	3.2s	151.6s	98%
Apriel-1.6-15B	Dense	188.9s	4.6s	193.5s	98%

Decode is between 1% and 6% of the time you actually wait.

If your model is answering questions, calling tools, writing code snippets, summarizing documents or doing anything that does not generate 10,000+ tokens of output, generation speed does not matter. At all. You can halve the decode speed of a model and no human will notice the difference. You can double it and no human will notice.

All benchmark reports that lead with tg128 are actively misleading for real world usage. The only number that matters is pp65K — pure prefill speed at 65,000 tokens context.

KV head count is the most important number no one checks

Parameter count does not predict long context performance. MoE vs dense does not predict long context performance.

The single strongest predictor of prefill speed at 65K+ context is n_kv_heads multiplied by head_dim from the model config. This is the size of the KV cache per token.

Model	Size	KV config	Bytes per token	pp131K speed	Slowdown from 4K
Ornith-9B	9B	4 × 128	64 KB	873 t/s	2.5×
Qwen3.6-27B	27B	4 × 256	128 KB	285 t/s	2.4×
Apriel-15B	15B	8 × 128	160 KB	197 t/s	6.1×
Granite-4.1-8B	8B	8 × 128	160 KB	244 t/s	7.4×

Ornith 9B is 4.4x faster at 128K context than Apriel 15B despite being 40% smaller. The entire difference is KV cache architecture.

This is not a small effect. Two models from the exact same family can differ by 3-4x at long context if one was released with 4 KV heads and the other with 8. No one lists this number on model cards. No benchmark summary includes it. You have to open the config.json and check for yourself.

The KV quantization paradox

Every guide will tell you to always quantize your KV cache to Q8 or Q4 for maximum speed. This was correct. It is no longer correct at long context.

At 65K context the dequantization cost for quantized KV cache exceeds the bandwidth saved by making the cache smaller. For MoE models and small dense models, unquantized F16 KV cache is faster than quantized:

Model	Type	Q8 KV speed	F16 KV speed	F16 advantage
Gemma-4-26B-A4B	MoE	1015 t/s	1554 t/s	+53%
Qwen3.6-35B-A3B	MoE	1273 t/s	1573 t/s	+24%
Ornith-9B	Dense	1276 t/s	1544 t/s	+21%
Trinity-Mini	MoE	1429 t/s	1583 t/s	+11%
Apriel-15B	Dense 8KV	351 t/s	120 t/s	-66%

F16 wins for MoE models. It loses catastrophically for dense models with 8+ KV heads.

There is no universal correct setting. Test F16 at your actual working context length. The 3 year old rule of always quantizing KV cache breaks completely above 32K context.

LongCat 2.0 lands: 1.6T open MoE under MIT license

Meituan dropped LongCat 2.0 this week. It is the first frontier scale model released under a fully permissive open license.

1.6T total parameters. 48B active per token. Trained on 35T tokens. Beats GPT-5.5 on SWE-bench Pro. Works out of the box with vLLM, SGLang and llama.cpp.

This is not a research toy. This model will be running production agent workloads before the end of the month.

LongCat introduces three architectural changes to attention that every other model will copy within 6 months:

1. Streaming aware indexing that delivers coalesced HBM access
2. Cross layer indexing that runs one attention index pass every 2 layers
3. Hierarchical coarse to fine token selection that cuts indexing cost by 70%

All three changes are fully backwards compatible with existing inference runtimes. No custom hardware required.

Practical deployment numbers for 2026 hardware

This is what you can actually run right now:

• 9B dense: 128K context on 16GB VRAM
• 27B dense: 105K context on 32GB VRAM
• 35B MoE: 128K context on 24GB VRAM
• 1.6T LongCat 2.0: 1M context on 16x H20

The Qwen 3.6 27B results are particularly notable. Running Q8 model weight, Q8 K cache, Q5 V cache you can hit stable 120 tokens per second aggregate at 105,000 context on a single RTX 5090. This was not possible 30 days ago.

Mamba2 and MLA: what actually delivers

Mamba2 is not hype. The architecture delivers exactly what was promised.

Granite 4.0 H Small retains 69% of its 4K prefill speed at 131K context. Every transformer model in the test dropped below 42%. At 128K context this 17GB Mamba2 hybrid ties the 6GB Ornith 9B on prefill speed.

It is currently let down entirely by training quality, not architecture. Someone will train a good model on this architecture this year. When they do it will obsolete every existing transformer for long context work.

MLA on the other hand does not deliver on Vulkan. GLM 4.7 Flash lost 80% of its prefill speed between 512 and 16K context and crashed entirely above that. This may be different on CUDA. Do not extrapolate short context MLA benchmarks.

The composite score: speed × intelligence

Raw speed is useless if the model can not do the work. Raw intelligence is useless if you spend three minutes waiting for it to respond.

This composite score combines model intelligence rating with inverse wall clock time at 65K context. This is the number you should be using to pick models:

Rank	Model	Type	Composite score
1	Qwen3.6-35B-A3B	MoE	29.4
2	Trinity-Mini	MoE	24.9
3	Gemma-4-26B-A4B	MoE	19.6
4	North-Mini-Code	MoE	14.0
5	Ornith-9B	Dense	13.7
6	Qwen3.6-27B	Dense	10.8

The single smartest dense model tested comes 6th. It delivers 87% of the intelligence at 1/3rd the speed. For agentic workflows that run 10+ iterations per task this is not even a tradeoff. MoE wins.

Actionable deployment rules

These are the rules you should follow today, not the ones you read last year:

1. Stop looking at tg128. Benchmark pp65K / pp131K instead
2. Check n_kv_heads and head_dim before you look at parameter count
3. Test F16 KV cache at your working context length
4. For consumer VRAM < 48GB always pick MoE over dense
5. Do not run any model with 8 KV heads for workloads over 32K context
6. Mamba2 hybrids are now worth evaluating for pure ingestion workloads

None of this is theoretical. All of these numbers were measured on real hardware by independent engineers this week.

Closing

We went through an entire phase of LLM development where generation was the bottleneck. That phase ended. We are now in the prefill era.

Almost all of the optimization work, benchmarking and discussion is still fighting the last war. People are arguing over 10% differences in generation speed while ignoring 400% differences in prefill speed.

This will shift over the next 6 months. For now you have a very large advantage if you just measure the right thing.