Skip to content

Open Weight LLMs Just Beat Proprietary Models, And You Can Run Them Locally

#local-llm #deepseek-v4 #longcat #quantization #llama.cpp

This is not a preview. This is not an announcement. Right now you can download model weights that perform within 1-2% of GPT-5.6 Luna and Claude Opus 4.8, run them on hardware you already own, and modify them for any purpose with zero restrictions.

That is the takeaway. Everything else below is implementation details.

What just happened this week

Over 72 hours starting July 30 2026 three separate drops reset every baseline for open LLMs.

DeepSeek released the final GA build of V4 Flash. Unsloth published properly calibrated GGUF quants 12 hours later. Meituan dropped LongCat Flash Lite Sparse with almost no fanfare. And an independent developer ported TurboFieldfare to run 35B MoE models in 1.4 gigabytes of system RAM.

None of these are research previews. All weights are available right now. All are MIT licensed.

DeepSeek V4 Flash 0731: The numbers that matter

Ignore the marketing. This model beats every open model ever released on every agent and coding benchmark. It is competitive with every closed model except the absolute current frontier.

This is the official benchmark table published with the release:

BenchmarkDeepSeek V4 Flash 0731GLM 5.2Opus 4.8
Terminal Bench 2.182.781.085.0
NL2Repo54.248.969.7
Cybergym76.7-83.1
DeepSWE54.446.258.0
Toolathlon Verified70.359.976.2
Agents Last Exam25.223.825.7
AutomationBench Public25.112.927.2

It scores 50 on the Artificial Analysis Index. That is one point below GLM 5.2 and GPT-5.6 Luna. It ranks exactly equal to Sonnet 5 and Grok 4.5 on DeepSWE.

Most importantly: it did this with 12B activated parameters. Not 400B. Not 200B. 12 billion parameters per forward pass.

This is not a bigger model. This is a better model.

Unsloth quantization: Why this changed everything overnight

12 hours after DeepSeek posted the base weights, Unsloth published their GGUF quant builds. This is the part almost no one is talking about.

Unsloth Dynamic 2.0 Q4 quant for this model is 155GB. The Q8 quant is 162GB.

That is 7 gigabytes difference between effectively lossless 8 bit and 4 bit quantization. No one has ever pulled that off before. For reference, standard K quants for a model this size would have a 70GB gap between Q4 and Q8.

You get 99.8% of full precision performance for 7 extra gigabytes over the 4 bit quant. This breaks every prior tradeoff curve for quantization.

For deployment this means you can run the full lossless model on a single H100 180GB. You can run the Q4 build on a single RTX 4090 24GB with partial offload.

Deployment cheat sheet for DeepSeek V4

Every single one of these commands works right now. No API keys. No waiting lists.

For a local server with built in web UI, one command:

bash
llama serve -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

For Ollama:

bash
ollama run hf.co/unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL

For Unsloth Studio:

bash
curl -fsSL https://unsloth.ai/install.sh | sh
unsloth studio

This is not beta software. This works. You will get 35-45 tok/s on an RTX 4090.

LongCat Flash Lite Sparse: The quiet drop no one talked about

While everyone was arguing about DeepSeek, Meituan released LongCat Flash Lite Sparse. It is MIT licensed. It has native 1 million token context. It runs on a single H20.

This model uses 3B activated parameters out of 69B total. It beats all prior open models on long context retrieval. Most importantly it implements the first actually good sparse attention implementation that does not degrade model quality.

Sparse attention done correctly

Everyone has been building sparse attention for 5 years. All prior implementations lost 5-10% benchmark performance. LongCat fixed this with three very simple changes:

  1. Streaming Aware Indexing reserves 30% of the attention budget for fixed sink and sliding window tokens. No more random missing context.
  2. Cross Layer Indexing reuses attention selection across 4 consecutive layers. This cuts indexing overhead by 75%.
  3. Hierarchical Indexing runs coarse block selection first, then fine token selection only inside matching blocks.

The end result: 99% of dense model quality at 30% of the inference cost for long context.

BenchmarkLongCat DenseLongCat Sparse
SWE-Bench Verified54.468.2
MMLU85.585.3
LongBench v247.153.6

Sparse attention no longer has a quality penalty. That is a permanent shift.

TurboFieldfare: Running 35B MoE in 1.4GB RAM

This is the most insane thing that happened this week and almost nobody saw it.

An independent developer ported TurboFieldfare to Qwen 3.6 35B. This engine does not load the entire model into memory. It streams MoE experts on demand directly from SSD.

It runs the full 35B model with 1.4 gigabytes of resident RAM. It produces byte identical output to the full loaded model. It runs at 22 tokens per second on an M5 Mac.

You can cap the process to 8GB working set and there is zero performance change. It does not care. All of the model lives on disk.

This breaks every assumption we had about LLM deployment requirements. You do not need VRAM. You do not need lots of RAM. You just need a fast SSD.

We are now past the proprietary moat

There is no longer any capability that only closed API models can deliver.

The gap between the absolute best closed model and the best open model is now 2% on aggregate benchmarks. That gap will be gone in 30 days.

Every single one of these models is MIT licensed. You can fine tune them. You can quantize them. You can run them on your own hardware. You can sell access to them. You can do literally anything.

Cloud API providers will not tell you this. They will continue to charge 100x the actual inference cost for as long as people will pay it.

The unspoken problems

None of this is perfect. There are real tradeoffs.

DeepSeek V4 Flash has extremely verbose reasoning traces. It will shout OH MY GOD in the middle of chain of thought. It gets excited. It makes jokes. Some people hate this.

LongCat has not been tested for safety alignment. It will do exactly what you ask it to do. No guardrails. No refusals.

TurboFieldfare only works on Mac right now. The Linux port is 2 weeks away.

All of these models will hallucinate. All of them will make mistakes. None of them are magic. They are just very good.

What comes next

DeepSeek has already confirmed V4 Pro will be released open weight within two weeks. That model is expected to match Opus 4.8 exactly.

Every optimization that was used for these models will be backported to every existing open model. Quantization will get better. Sparse attention will become standard. SSD streaming will be supported in every runtime by the end of the quarter.

This is not a peak. This is the baseline.

Six months from now we will look back and laugh that anyone ever paid for LLM API access.