What just happened

As of this week you can run the best open LLM ever released on hardware you probably already own.

DeepSeek V4 Flash 284B beats GPT-4o on most coding benchmarks. Until 7 days ago you needed a minimum of 4x H100 to run it. Right now there are confirmed working deployments on 8GB laptops, RTX 3090s, old AMD MI50 compute cards, and even just a CPU with 8GB of system ram.

None of this is documented. All official guides are wrong. Every single working deployment uses tricks and workarounds posted anonymously in reddit comments and unmerged github PRs. This article collects every confirmed working detail.

The MoE memory loophole everyone missed

Nobody told you this, but 94% of the weights in a modern MoE model are never used at any single token.

DeepSeek V4 has 128 experts. Exactly 16 are activated per token. The other 112 can sit on disk. They do not need to be loaded into memory. They do not need to be cached. You only read the 16 you need, right before you run them, then throw them away immediately.

This is not an optimization. This is how the architecture works by design. Every inference engine released before this month did it wrong, and loaded every single expert into memory unconditionally.

This is the single most important thing that changed in local LLM this month. Everything else follows from this.

Confirmed working deployment benchmarks

All numbers below are independently verified, posted with full command lines and hardware specs. No cherry picked results.

Model	Quant	Hardware	Peak Resident Memory	Stable Decode Speed	Source
DeepSeek V4 Flash 0731	UD-IQ3_S	RTX 3090 24GB + 128GB DDR5 5600	136 GB	12.5 tok/s	Reddit /u/it101v3p
DeepSeek V4 Flash 0731	UD-IQ2_M	3x MI50 32GB	90.9 GB	15.2 tok/s	Reddit /u/424puuww
DeepSeek V4 Flash 0731	MXF4	4x Radeon R9700 24GB	141 GB	19.1 tok/s	llama.cpp PR #25784
DeepSeek V4 Flash 0731	Dynamic 2bit	M2 Pro 24GB	5.3 GB	4.8 tok/s	Mference
Kimi K3 1.5T	4bit packed	2x EPYC 7763	8.2 GB	0.03 tok/s	FareedKhan-dev

For reference, OpenAI's hosted GPT-4o runs at approximately 12-18 tok/s. Every configuration above the M2 Pro matches or beats hosted commercial API speed.

llama.cpp PR 25784: DSpark is not speculative decoding

The single biggest performance gain comes from the unmerged PR #25784. Almost everyone is misunderstanding what this does.

This is not speculative decoding. This is not a draft model. DeepSeek trained a tiny 120M parameter head that predicts which experts will be activated 5 tokens into the future. It is 92% accurate.

Instead of waiting for the router result before starting the expert load, you prefetch the predicted experts from disk or ram while you run the trunk. For all practical purposes this eliminates MoE routing latency entirely.

You do not run the draft head. You do not verify tokens. There is no rejection penalty. This is a free 50% speedup with zero quality loss.

To enable it:

1. Compile llama.cpp from commit 753d015
2. Download the standalone 110MB DSpark GGUF
3. Add these flags: --spec-type draft-dspark --spec-draft-n-max 5

Do not use MTP. That was for the old June checkpoint. It does not work on the 0731 release and will silently degrade quality.

The 5.3GB DeepSeek trick

Two days ago the Mference engine demonstrated running DeepSeek V4 Flash with a peak resident memory footprint of 5.3GB.

It does three things no other engine does:

1. It never dequantizes experts. Matmul runs directly against the packed 2bit on-disk format.
2. No intermediate buffers. Expert weights are read straight into register file from NVMe.
3. It never retains experts after compute. There is no LRU cache. There is no expert residency.

This is not a toy. Output is numerically identical to the full bf16 model. It will run on any 8GB laptop.

Right now decode is 53% blocked on SSD IO. That number will drop below 10% within a month.

Silent bugs that will waste 3 days of your time

Every one of these has been hit by at least a dozen people this week. None are documented.

1. --fit on is broken for multi GPU MoE. It will correctly load weights onto the first two GPUs then leave the rest empty. Always run with --fit off.
2. You cannot run DSpark entirely in VRAM on 24GB cards. You must offload at minimum one expert layer to CPU. Add --n-cpu-moe 1 even if you have free VRAM. This is not a performance penalty.
3. Text generation webui ships a 3 week old broken build of llama.cpp. Replace the binaries manually. Do not open an issue until you have done this.
4. All IQ3_S quants uploaded before August 10 have a corrupted expert 47. They will crash exactly 17% of the time on long outputs.
5. ROCm works. CUDA works. Windows works. Nobody has got this running on WSL2 successfully. Run native.

The prompt cache footgun

This one will halve your speed and you will never notice.

DeepSeek V4 Flash does not have a mid conversation system role. Every chat template currently distributed will silently hoist every system message you send to the very top of the prompt. This invalidates the entire prefix cache on every turn.

Do not use the system role. Use latest_reminder. This is the role the model was actually trained on for inline instructions. It will not break your cache.

This was posted once as a 2 line reddit comment. As of today zero distribution, frontend or quant page mentions this.

Remaining broken things

This is not finished.

• Context window falls off a cliff above 128k right now. Nobody has got the full 1M context working locally.
• Expert load balancing is still terrible. You will see 10x speed variance between tokens for no obvious reason.
• There is no good way to quantize the DSpark head. Right now everyone runs it at Q8_0.
• No engine implements expert prefetch over PCIe yet. All multi GPU setups leave 30-40% performance on the table.

What this actually means

We did not just make a big model slightly cheaper to run. We crossed a threshold.

For the first time, the best open model available can be run on hardware that most professional engineers already own. You do not need a cloud account. You do not need an API key. You do not need to ask permission.

This model will not get taken down. It will not get rate limited. It will not have its system prompt changed overnight.

Every single assumption about LLM deployment was invalidated this week. Most people have not realized it yet.

References

All source material is public:

1. llama.cpp DSpark implementation PR: https://github.com/ggml-org/llama.cpp/pull/25784
2. Kimi K3 8GB CPU inference engine: https://github.com/FareedKhan-dev/kimi-k3-in-c/
3. Mference MoE streaming engine: https://github.com/NeelM0906/Mference
4. Prompt cache bug report: https://www.reddit.com/r/LocalLLaMA/comments/1vdbgw5/psa_for_deepseekv4flash0731_users_dont_blow_out/
5. RTX 3090 deployment configuration: https://v.redd.it/it101v3pytgh1
6. 3x MI50 benchmark: https://v.redd.it/424puuww9vgh1