Bonsai 27B changed the rules for edge AI this week.

Until 5 days ago, if you wanted a 27B class model you needed 18GB of VRAM minimum. Today you can run one on an iPhone. You can run one in a Chrome tab.

This is not another incremental quantization trick. This is the first time anyone has demonstrated that you can throw away 93% of the bits of a modern LLM and retain almost all of its reasoning ability.

What actually shipped

Two variants were released, both fine tuned from Qwen3.6 27B, both published under Apache 2.0 license with no commercial restrictions:

Variant	Effective bits per weight	Total on disk size	Full precision performance retention	Target hardware
1-bit Bonsai 27B	1.125	3.9 GB	89.5%	Phones, 4GB VRAM laptops
Ternary Bonsai 27B	1.71	5.9 GB	94.6%	8GB VRAM laptops, desktops

Both variants include the full 262K context window, native structured tool calling, and an integrated 4-bit vision tower. No layers are kept at higher precision. The entire transformer runs end to end at the target bit depth with no escape hatches.

Performance breakdown by capability

Everyone quotes the average retention number. The important detail is where the performance loss occurs.

Loss is almost entirely concentrated in creative writing and trivial factual recall. Reasoning, coding and tool calling are almost untouched. That is the exact opposite result of every prior quantization method. All previous low bit builds destroyed reasoning first. This one preserves it.

Runtime support status

This is not a locked research demo. It works today on almost every runtime people actually use for local LLMs.

Runtime	1-bit Bonsai support	Ternary Bonsai support
llama.cpp CPU	✅ Merged upstream	✅ Merged upstream
llama.cpp Metal	✅ Merged upstream	✅ Merged upstream
llama.cpp CUDA	✅ Merged upstream	🟡 PR under review
llama.cpp Vulkan	✅ Merged upstream	🟡 PR approved pending merge
MLX	🟡 PR open	❌ Not yet implemented
WebGPU	✅ Community kernel released	❌ Not yet implemented

All open PRs are expected to land within 7 days. By the end of this month every major local LLM runtime will support both variants out of the box.

It runs in the browser

Three days after the model release, the webml community shipped hand written WebGPU kernels that run the 1-bit variant entirely client side. No server. No permanent download. Just open the tab and it runs.

Right now it hits ~18 tokens per second on a mid range laptop GPU. That is fast enough for real time chat.

This is the single most underreported part of the release. For the first time you can embed a capable general purpose LLM directly into a web page, with zero infrastructure cost.

Apple is already talking to PrismML

CNBC confirmed Apple is in active talks with PrismML for integration into iOS 19. This should surprise nobody.

The 1-bit variant fits cleanly inside the 6GB per app memory limit on iPhone 17 Pro. No other 27B class model comes within a factor of two of that budget. Apple spent the last year demoing 7B and 14B models on device. They can now ship a 27B.

This is not a hypothetical. The MLX PR for 1-bit support was opened the same day the model launched.

Documented limitations

The maintainers have been unusually honest about what this model does not do well.

Agentic coding workflows are bad. Not a little bad, noticeably worse than baseline Qwen3.6. This is not a quantization artifact, this variant was not tuned for long horizon run-test-and-repair loops. A dedicated agent fine tune is scheduled for release next month.

Ternary does not fit on phones. It comes in at 7.2GB with current packing, just over the iOS hard limit. Native ternary kernels will bring this down to 5.9GB, but that work is not complete.

KV cache is still 4 bit. The model tolerates much higher compression here. Early testing shows sub 2 bit KV cache works with negligible quality loss, which would double effective context length on edge devices.

How this quantization actually works

Most quantization works by rounding individual weights and hoping the error cancels out. Bonsai does the opposite.

They did not quantize weights. They trained the rounding. Every weight value is chosen to minimize error on the final model output, not minimize error on the individual weight. That is the entire trick. Nobody had done this properly at 27B scale before.

Real world observed performance

Independent user benchmarks posted this week match the published numbers almost exactly:

• RTX 4060 8GB: 30 tokens/sec 1-bit, 22 tokens/sec ternary
• M3 Pro: 47 tokens/sec 1-bit, 31 tokens/sec ternary
• M5 Max: 87 tokens/sec 1-bit, 58 tokens/sec ternary
• RTX 5090: 163 tokens/sec 1-bit

For reference, this is the same speed most people were getting from 7B models 12 months ago.

What this changes

We have crossed a threshold.

For the last three years the entire field operated under an unstated assumption: 4 bits was the practical minimum for usable models. Everyone optimized around that line. Bonsai just moved that line by 75%.

This is not the end of the line. This method works on every model architecture. It works on 70B models. It works on multimodal models. It works on diffusion models.

Raw model capability will keep improving. But for the next 12 months, intelligence density will be the axis that actually changes what people build. You can already run a model that would have been state of the art 18 months ago in your pocket. In six months you will run one that is state of the art today.

This release will not kill cloud models. Frontier models will always be better. But 90% of the things people actually use LLMs for every day do not need a frontier model. They just need something good enough, that runs when you have no signal, that never sends your data anywhere.

Bonsai 27B is that thing.