Six months ago you needed a 24GB RTX 3090 to run a good reasoning model at usable speed. Three months ago that bar dropped to 16GB for 14B models. This month, independent developers broke every existing expectation.

Right now you can run Qwen 3.6 27B at 40 tok/s entirely in VRAM on a $300 RTX 5060 Ti 16GB. You can run the 35B MoE variant at 30 tok/s on an 8GB 3070 Ti.

That is not a typo. None of this uses cloud instances, none of this requires proprietary software. All of this works with open source models and modified llama.cpp runtimes.

The quiet inflection point

Nobody at the big vendors announced this. There was no press release, no paper, no official benchmark. Every single one of these breakthroughs came from random users posting on /r/LocalLLaMA, uploading quantized models to Hugging Face, and forking llama.cpp.

This is not incremental improvement. This is a step change. For the first time, production grade reasoning models fit and run well on hardware that normal people already own. There is no longer any technical reason most internal business LLM workloads cannot run completely on premise on existing server or desktop hardware.

We have already passed the point where hosted commercial APIs have any inherent performance or quality advantage for 90% of use cases. They only retain advantage for convenience.

Pure quantization: removing the fat

The first big breakthrough this month is pure quantization.

Standard GGUF quantizations include a lot of overhead that almost no one needs: unused expert weights, dead alignment padding, metadata, unused output heads and debug tensors. For months everyone just accepted this as unavoidable. Then two weeks ago a user posted a modified quant script that strips all unused weight ranges before quantization runs.

For Qwen 3.6 27B Q4_K_M, this reduces total model size from 17.1GB (official Unsloth quant) down to 15.1GB. That 2GB difference is the entire gap between fitting entirely in 16GB VRAM and spilling over to slow system RAM.

At 15.1GB you can run this model with 65k context, no offload, no swapping. The MTP speculative variant comes in at 15.4GB and hits 40 tok/s token generation. Perplexity degradation is measured at 0.12% relative to the full weight model. No one can tell the difference in blind testing.

This is not a trick. This is just deleting data that was shipped in the model file but never executed at runtime. Every major quant uploader is already switching to this method. As of this week, any quant you see that does not say 'pure' is 10-15% larger than it needs to be.

DFlash: speculative decoding that actually works

Everyone has known for two years that speculative decoding can give 2-3x speedups. Everyone also knew that in practice it almost never delivered that, broke on reasoning, had terrible acceptance rates, and fell apart with long context.

BeeLlama 0.2.0 fixed this last week.

On a single RTX 3090, DFlash runs Qwen 3.6 27B at 164 tok/s. That is 4.4x faster than baseline llama.cpp. Acceptance rate sits at 67.7% for general output, 89.2% for code. Most importantly, it does not degrade output quality. It correctly preserves reasoning blocks, tool calls, and structured output.

This is not overfit to benchmarks. Multiple independent users have confirmed these numbers on real workloads. For the first time, speculative decoding is not a demo trick. It is something you can turn on for production use.

The implementation required no changes to the base model. All improvements came purely from better draft validation logic and KV cache reuse. This will work for every modern LLM, not just Qwen.

MoE quantization hacks everyone is sleeping on

Almost everyone still gets MoE deployment wrong.

The standard advice is that you must fit the entire MoE model into VRAM for good performance. That is wrong. Qwen 3.6 35B A3B only activates 3.5B parameters per forward pass. You only need those 3.5B parameters, plus KV cache, on the GPU at any time. The rest of the experts can sit on system RAM with effectively zero performance penalty.

This is how you get 30 tok/s and 262k context on an 8GB 3070 Ti. 3GB for active layers, 2GB for runtime buffers, 2.56GB for Q8 KV cache. It fits exactly.

Nearly all public benchmarks are still loading every expert into VRAM. They are wasting 80% of their VRAM for no gain. If you are running an MoE model today and you are not doing partial expert loading, you are leaving 2-3x performance on the table.

You also get an extra 25% speedup just by booting Ubuntu Server instead of Windows 11. Windows reserves 1.2GB of VRAM by default for desktop compositor garbage. That 1.2GB is exactly the difference between fitting the model and not fitting it.

The quantization quality tradeoff nobody talks about

There is now a very clear, measurable tradeoff that no benchmark will tell you.

All modern 4 bit quants are good enough that you will not notice quality difference in normal chat. The difference between them shows up only in failure modes.

Bad quants do not give slightly worse answers. They give blank outputs. They silently drop context after 40k tokens. They break regex structured output. They fail search and replace operations. They will pass every standard perplexity test and then fail silently at 2AM on your production workload.

This is why the new IQ4_KS quant from ik_llama.cpp is important. It has identical perplexity to IQ4_XS, but completely eliminates silent failure modes. It runs 1.7x faster, and has run for 7 days straight in production use with zero bad outputs.

You will not see this difference on MMLU. You will only see it after 100 hours of real use.

What you should run right now

This is the current state of the art as of May 26 2026. All entries have independent third party confirmation:

GPU VRAM	Model	Quant	Token generation speed	Maximum stable context
6GB	Qwen 3.6 35B A3B	ByteShape Q4_K_S	22 tok/s	32k
8GB	Qwen 3.6 35B A3B	IQ4_NL_XL	30 tok/s	262k
16GB	Qwen 3.6 27B MTP	Pure Q4_K_M	40 tok/s	65k
16GB	Qwen 3.6 27B	IQ4_KS	28 tok/s	105k
24GB	Qwen 3.6 27B	DFlash Q4_K_M	164 tok/s	128k

All of these work today. None require anything other than llama.cpp or one of its public forks.

Closing observations

None of this work came from OpenAI, DeepSeek, Meta, Unsloth or any of the funded companies. Every single one of these breakthroughs was built by random independent developers, posting for free on an internet forum.

That is not an accident. The big organizations are all optimizing for datacenter deployments on 80GB H100s. No one at those companies is trying to make a model fit on a 16GB consumer GPU. No one there even owns one.

This will keep happening. The gap between what is officially supported and what you can actually build will keep growing. Right now, if you are willing to spend an afternoon reading forum threads and tuning launch parameters, you can build a local LLM deployment that outperforms most commercial hosted APIs, on hardware you already own.