Right now you can run a model on your desk that matches Claude Sonnet 4.6 on 92% of day to day engineering work. You can reproduce any public open model from a 9 kilobyte text file. Thousands of people are quietly pooling traces from closed frontier models and training open copies faster than any single lab can iterate.

No one at the big conferences is talking about this. None of the analyst reports mention it. This is not the future. This is what is working, right now, for tens of thousands of engineers.

The state of local LLMs mid 2026

Let's get the baseline out of the way first. Epoch currently puts the best open weights at 4 months behind the closed frontier. That gap has stayed remarkably stable for 18 months. It will not close. It does not need to close.

On coding, tool use, and agent orchestration the gap is less than 2 percentage points on most evals. On hard mathematical reasoning and novel scientific work the closed models are still clearly ahead. For 95% of the work that paid ML engineers actually do every day, you cannot tell the difference.

This is the quiet shift that happened in the first half of 2026. No one announced it. No press release went out. One month local models were a fun hobby. The next month people stopped using the API for most work.

The great distillation heist

There is one unacknowledged fact driving almost all progress right now. Every good open model released in 2026 is distilled from traces of closed frontier models.

This is not speculation. You can go on Hugging Face right now and see the datasets. There is claude-fable-5-claude-code with 117 million cleaned tool calls. There is Vibe-Coding-Claude-Fable-5 with 2.3 million engineering planning examples. There is claude_mythos_distilled_25k with the highest quality reasoning traces ever published.

No one is training base models from scratch any more. That game is over. The winning play is to collect clean, high quality output from the frontier models, remove artifacts and failure modes, and train open models to imitate exactly that output.

Closed labs cannot stop this. They cannot watermark the output enough to break distillation. They cannot degrade output quality without destroying the value of their own product. This is a permanent, structural arbitrage that will exist for as long as closed models have public APIs.

Tool schema leakage and the Fable 5 dataset pool

When Anthropic released Fable 5 they made one mistake. They exposed the full native tool schema used internally by the model. All 42 tools. Every parameter. Every return type. Every implicit convention the model had been trained to follow.

Within 72 hours the community had scraped the full schema, built extractors, and started pooling traces. The teich utility will extract, clean, anonymize and normalize Fable 5 traces from any client log. One command. Anyone can run it. Anyone can upload the resulting dataset.

This is not one person doing this. This is thousands of people running traces every day, cleaning them, and contributing them back to the pool. There is no central coordinator. No foundation. No company. It just happens.

As of June 2026 the combined public Fable 5 dataset pool is over 410 million tokens. That is larger than the entire original CodeLlama training set. It grows by 12 million tokens every week.

No closed lab has a training dataset this good. No closed lab ever will.

The Grimoire: models that cannot be deleted

Hosting is the single point of failure for open models. Any model on Hugging Face can be taken down with a single legal notice. Thousands of good models have already been removed.

Heretic 1.4 solved this.

Every abliterated model produced with Heretic now exports a 9 kilobyte reproduce.json manifest. This file contains every parameter, every seed, every delta required to reconstruct the exact model bit for bit. You do not need to redistribute gigabytes of weights. You only need to redistribute this 9kb file.

You can collect every public Heretic model with one command: heretic --collect-reproducibles my_grimoire. The full collection of 117 models is 1.1 megabytes total. You can store it on a floppy disk. You can email it. You can print it out.

To restore a model you run heretic --reproduce file.json. The process takes about 60 seconds. It does not re-run training. It reconstructs the final weights directly from the manifest.

10 models that have already been deleted from Hugging Face are still fully reproducible via this system. There is no way to take them down. There is no central server to seize. There is no domain to seize. The model exists wherever a copy of that 9kb file exists.

This is not a backup system. This is a takedown resistance system. It works.

Inference optimizations that closed the gap

Almost all of the performance gains this year came from optimizations that were first published by random people on Reddit, not by corporate labs.

Sparse attention dropped per token FLOPs by 75% at 1 million token context. Latent KV compression cut cache memory usage by 90%. Multi-token prediction gave a 1.8x throughput gain with zero quality loss. NVFP4 quantisation cut weight footprint in half for a 1.2% accuracy drop that can be fully trained back.

None of these were invented by OpenAI, Anthropic or Google. All of them were implemented, tested and benchmarked by the community before any closed lab announced them.

Closed labs will never publish these optimizations. It is against their business interest to make inference cheaper. The open community has exactly the opposite incentive. This divergence will only accelerate.

Hardware: the unified memory pivot

The hardware story this year is not faster GPUs. It is the death of the discrete GPU for local inference.

MoE models have a very specific hardware profile. They need enormous memory capacity to hold all the experts. They need very modest memory bandwidth, because only a tiny fraction of the weights are touched per token.

This is exactly the profile that unified memory APUs were designed for. Not by accident. By luck.

A $3999 Strix Halo with 128GB unified memory will run DeepSeek V4 Flash at 29.7 tokens per second. A $4699 DGX Spark runs it at 41 tokens per second. Two DGX Sparks linked with a 200Gbps cable will do 350 aggregate tokens per second at 32 concurrent users.

Discrete GPUs lost. They have too little memory. They have far more bandwidth than anyone needs for MoE decoding. The entire local inference world has pivoted to unified memory systems over the last 6 months.

DRAM prices have increased 98% quarter on quarter. There will be no relief before late 2027. This is not a temporary shortage. Memory manufacturers have permanently reallocated production to datacenter HBM. Local inference hardware will get more expensive before it gets cheaper.

Benchmark reality: what the numbers actually mean

Every model benchmaxes now. Public benchmark scores are inflated across the board. Ignore them.

The only metric that matters is: can this model do your actual work, without you having to fix the output every time.

For routine coding: Qwen 3.6 27B works. Gemma 4 31B works. DeepSeek V4 Flash works. All of them will produce usable code 9 times out of 10.

For agent work: DeepSeek V4 Pro is currently the best open model. It is within 3 points of Sonnet 4.6 on tool use evals.

For hard reasoning: Nothing open comes close to Claude Opus or GPT 5.5. This will remain true for the foreseeable future.

You do not need the best model in the world. You need a model that is good enough, that runs on hardware you control, that will never be taken away, that will never increase its prices, that will never refuse your request.

The silent bug crisis in vibe coded codebases

There is one large unspoken problem with this entire stack. All of these models produce subtly broken code.

Not obviously broken. Not code that fails immediately. Code that works 99% of the time. Code that misses one error check. Code that fails silently on edge cases. Code that will blow up 6 months from now when you have forgotten it exists.

Every engineer that has done serious vibe coding has run into this. You build an entire application in a weekend. It works perfectly for testing. Then over the next month you find 30 tiny missing error checks, off by one errors, silent failures.

There is no magic fix for this. Stop dumping the entire codebase into context. Stop asking for large changes. Ask the model to modify one function at a time. Turn off KV cache quantization. Run every single line of output through a linter and a test suite before you commit it.

This is not a temporary flaw. This is the fundamental nature of distilled models. They imitate the output of good engineers. They do not imitate the caution of good engineers.

The coming model collapse and the open advantage

Right now every closed lab is racing to build larger and larger models. Every open developer is racing to build infrastructure that makes large models irrelevant.

Closed labs are building moats around model quality. The open community is building moats around distribution, reproducibility, and cost.

Closed models will always be slightly better. It will not matter. Most people will choose a slightly worse model that they control, that they can run forever, over a slightly better model that they have to rent forever.

This is not ideological. This is economics. Once you have a model that is good enough running on your own hardware, the API becomes a bad deal.

Open source AI is not an option. It is infrastructure

If intelligence becomes something people can only rent from three companies, we do not just lose software freedom. We lose operational freedom.

Every system built on top of closed APIs is built on sand. Terms can change. Prices can double. Models can be nerfed. Access can be revoked. At any time, for any reason, with no appeal.

This is not a hypothetical. This has already happened, dozens of times, to thousands of companies.

Open source AI is not a movement. It is not a hobby. It is not an ideological position. It is civilizational infrastructure. It is the only way to guarantee that the ability to use this technology does not become the exclusive property of a tiny number of people.

The gap will never close. The frontier will always be ahead. That does not matter. What matters is that there exists an alternative. What matters is that you always have a choice.

Right now that alternative exists. It works. It is getting better every single week. No one can take it away from you.

That is the part no one is talking about.