The price gap just closed

A few months ago, using a cheaper Chinese model felt like a tradeoff. You saved money and got noticeably worse output. That gap is closing. In some cases it has closed entirely.

I've been running the same prompts through DeepSeek and a couple of Qwen variants against the APIs I was paying for. For summarizing customer feedback, drafting copy, and generating boilerplate, the difference is small enough that I'm having a hard time justifying the price difference. The cost compression is happening at the model layer, and that changes the math for anyone building on top of these APIs.

The harder part to reason about is trust and data handling. For a hobbyist project it barely matters. For anything touching user data it matters a lot, and the answers there are murky.

What's driving this shift is a release cadence that's hard to keep up with. Qwen 3.8 dropped open weights across three tiers. DeepSeek's V4 Flash is running on hardware people already own. And the tooling around these models, quantization, speculative decoding, even custom RISC-V chips, is maturing fast enough that "run it locally" is becoming the default answer instead of the compromise.

Key Numbers:

• 27B parameters fits in 32 GB of VRAM, so a single RTX 5090 or two 16 GB cards serve it without cloud GPUs.
• 99.4 tok/s prompt processing on a 144 GiB MoE model across four RTX 3060s, enough to ingest a 20k-token prompt in about three and a half minutes.
• 2.7-3.4× decode throughput with DFlash 2 speculative decoding on Qwen3.8-27B, turning a 10 tok/s decode into roughly 27-34 tok/s.
• 30 tokens per second on Alibaba's RISC-V XuanTie C950, with a 1.9 second time to first token.

What actually shipped in the Qwen 3.8 family

The release had three tiers, and only one of them matters for most local setups.

Qwen3.8-27B is the consumer sweet spot. 27 billion parameters, a native 262,144-token context window, and coding capabilities that reviewers have compared to Opus 4.5. It runs on a single MacBook or a 32 GB VRAM desktop. This is the model people are actually switching workflows to.

At the other end, Qwen3.8 2.4T Max is open weights in the literal sense only. Someone rented a B200 cluster and made a Call of Duty clone with it in one prompt, burning 1.1 million output tokens over five hours. Realistically barely anyone can run this locally, but its existence matters. The quantization community now has frontier-level weights to work with, and that tends to produce consumer-friendly derivatives within weeks.

A midsize model is due next week, per a Qwen community manager, and the speculation is that it lands over 100B parameters. If that happens, the gap between "local" and "frontier" narrows again.

The thing that trips people up with the 27B is the default reasoning effort. It ships with xhigh reasoning enabled, which means it "thinks" a lot out of the box. Reasoning traces eat context window, and responses take a while. Early takes called it slow, wasteful, an overthinker. Those takes are half right and half missing the point.

"Good enough" is doing a lot of work

Quick Take: the cost compression is happening at the model layer, and that changes the math for anyone building on top of these APIs.

The quality question is settled for practical tasks. Summarization, copy drafting, boilerplate generation, code review, bug hunting: the open-weight models are within noise of the paid APIs. The remaining question is operational, not qualitative.

Data handling is the real divider. If your prompts contain customer data, sending them to a foreign API is a decision your legal team gets a say in. Running the same model locally removes that conversation entirely. That's a big part of why local deployment is getting so much attention. It's cheaper, and it's cleaner from a compliance standpoint.

The community is split on how far to push this. I've seen people replace entire paid workflows with DeepSeek and Qwen, and I've seen people keep the paid APIs for anything touching user data while using local models for everything else. Both positions are defensible. What's no longer defensible is paying a 10x premium for output that's measurably equivalent on routine tasks.

A 144 GiB MoE on four 12 GB GPUs

The DeepSeek V4 Flash setup that's been making the rounds is a masterclass in squeezing blood from stone. The model is a 143-144 GiB MoE GGUF, quantized to UD-Q4_K_XL, running on four RTX 3060 12 GB cards with a 368k-token context window. Prompt processing hits 99.4 tok/s. Generation runs at 10.1 tok/s.

The GPU layout is the interesting part. The -ncmoe 34 flag keeps the experts from blocks 0-33 in system RAM. The remaining nine expert layers are explicitly distributed across GPUs 1-3, three layers per GPU. The extreme -ts 100,1,1,1 split doesn't distribute those explicitly assigned expert weights. Instead, it pushes most non-expert tensors, attention, KV-related allocations, everything else, onto GPU0. That leaves enough space on GPUs 1-3 for the large expert layers.

I tried to calculate this layout analytically and kept getting it wrong. Tensor placement with -ncmoe and explicit -ot overrides is discrete and unintuitive. The person who got this working measured every candidate configuration instead, and that's the approach that worked.

Microbatch size turned out to be the biggest performance lever. At -ub 1024, prompt processing ran at 63.4 tok/s. Bumping to -ub 2048 nearly doubled it to 99.4 tok/s. Decode stayed flat at 10.1-10.5 tok/s either way.

Configured context	Prefill	Decode	Min free VRAM
376,832	99.5 tok/s	10.4 tok/s	611 MiB
368,640	99.4 tok/s	10.1 tok/s	671 MiB
360,448	99.4 tok/s	10.1 tok/s	735 MiB

A few findings I've stolen for my own configs: Q8_0 KV cache is the safe default, F16 KV at 393k context leaves only 587 MiB free, and -np 1 is mandatory because multiple slots multiply KV-cache requirements. The model mostly lives in system RAM, so quad-channel memory bandwidth matters heavily. Even so, 100 tok/s prompt ingestion from a 144 GiB MoE model on four consumer 12 GB GPUs is much better than anyone expected.

Wall-clock time beats tokens per second

The Qwen3.8-27B launch produced a wave of hot takes about slow tokens and overthinking. Most of them miss the point. When you're doing agentic coding, the metric that matters is wall-clock time to a correct result, not tokens per second.

There are two modes of working with a local model. Mode one: high token throughput across multiple rounds, where the model fixes its own bugs and you stay in the loop guiding rework. Mode two: lower throughput, but the model works autonomously until completion while you go do something else.

For a given target quality, mode two wins even if it takes double the wall-clock time, because you context-switch less. A "slower" model that finishes unattended beats a faster model that keeps you in the loop. On a Ryzen 7 5700X with two RTX 5060 Ti 16 GB cards, Qwen3.8-27B tops out around 45 tok/s with MTP speculative decoding enabled. That's not impressive on paper. It's impressive because the model runs for an hour, produces a complete, correct result, and you weren't watching.

The serving config that works on 32 GB of VRAM goes against several pieces of folk wisdom. KV cache quantized to q4_0. MTP drafter KV cache quantized to q4_0 as well. "Friends don't let friends quantize the KV cache," one YouTube comment said. The empirical result: KV cache values are less sensitive to quantization than keys, and the tradeoff buys you the full 262k context window. The Qwen team recommends a hot sampling temperature of 1.0. Running it colder to fix "overthinking" is second-guessing the model's creators without evidence.

Speculative decoding is where the speed comes from

Speculative decoding is the optimization that matters for local inference. A small draft model guesses a block of tokens, and the target model verifies the whole block in one forward pass. Good guesses turn one pass into several tokens. Bad ones get thrown away.

For years the draft itself stayed autoregressive: one token at a time. DFlash made it one-pass, predicting the entire block in parallel. DFlash 2, released this week for Qwen3.8-27B and Meta's Muse Glimmer, adds two cheap fixes. A path selector scores adjacent candidate pairs to pick coherent sequences, costing 2.0M parameters and 0.6% latency. A two-tap dynamic convolution handles short-range dependencies within the block, costing 16.5M parameters and 0.7% latency.

The result is over 20% more output from every verification pass. On Qwen3.8-27B, SGLang serves at 2.7-3.4× the throughput of autoregressive decoding at batch size 1. The output is provably unchanged, which matters for anyone who's paranoid about speculative decoding changing model behavior.

The per-dataset numbers for the Qwen3.8-27B drafter tell the same story. DFlash 2 beats both the native MTP path and the community DSpark drafter on every benchmark.

Dataset	MTP	DSpark	DFlash 2
GSM8K	5.02	4.36	5.46
MATH-500	4.72	3.92	5.28
HumanEval	3.91	3.30	4.39
MBPP	3.99	3.51	4.79
MT-Bench	3.74	3.01	4.10
Mean	4.28	3.62	4.80

For local deployment, this changes the calculus. A 10 tok/s decode with DFlash 2 becomes effectively 27-34 tok/s. That's the difference between watching a progress bar and walking away from the machine.

RISC-V enters the chat

Alibaba's XuanTie C950 takes a different path to the same destination. It's a server-grade 64-bit RISC-V processor with 64 cores, clocks up to 3.2 GHz, and matrix and vector acceleration engines embedded directly in the silicon. No GPU needed. It runs Qwen3.8-27B natively at 30 tokens per second with a 1.9 second time to first token.

The C950 runs a single inference thread per socket, which makes it better suited for edge deployment and private inference than high-concurrency public APIs. That's a deliberate trade. The RISC-V ISA means no licensing fees to x86 or ARM, and Alibaba gets day-zero support for its own models on its own chips. It's the same vertical integration playbook NVIDIA ran, applied to open weights.

30 tok/s on dedicated hardware isn't going to replace a GPU server. But for a specific niche, a private appliance that runs your models with no cloud dependency and no per-token billing, it's a compelling option. The 27B model that runs on this chip is the same one running on 32 GB consumer VRAM, which means the software stack transfers directly.

Common Pitfalls

Treating xhigh reasoning as the default. Qwen3.8-27B ships with extra-high reasoning effort enabled. It burns context window and makes the model feel slow. For most tasks, lower reasoning effort produces the same result in a fraction of the time. Set it per task instead of accepting the default.

Believing speculative decoding is free. MTP drafters require VRAM and compute buffers. On a tight 32 GB setup, the drafter KV cache will eat memory as context fills. Quantize the drafter KV cache to q4_0. The quality impact is negligible and the memory savings are real.

Chasing tokens per second instead of wall-clock time. A model that finishes a task unattended beats one that keeps you in the loop, even at half the token rate. Benchmark against task completion, not decode speed.

Quantizing KV cache without knowing the tradeoff. KV cache values are less sensitive to quantization than keys. q4_0 values with q8_0 keys is a reasonable compromise. But watch what happens as context utilization grows. The tradeoff that works at 10k tokens may not hold at 200k.

Calculating MoE tensor placement analytically. With -ncmoe and explicit -ot overrides, placement is discrete and unintuitive. Measure every candidate configuration. The DeepSeek Flash setup that hit 99.4 tok/s came from empirical testing, not from theory.

One Thing to Remember

The models are changing faster than the hardware. Qwen 3.8 just dropped, a midsize model is due next week, and DFlash 2 drafters shipped the same week as the models they accelerate. Whatever configuration you settle on today will look outdated in a month. The cost of switching is a download and a config file.

The Bottom Line

If you're paying for API access for summarization, copy drafting, or boilerplate, switch to DeepSeek or Qwen now. The quality gap for practical tasks has closed, and the price difference is no longer justified. Keep the paid APIs only where data-handling requirements demand them.

If you have 32 GB of VRAM, run Qwen3.8-27B with a UD-Q4_K_XL quant and enable MTP or DFlash 2 speculative decoding. That combination approaches February-2026 frontier quality at home, and the serving config is well documented at this point.

If you're serving agents at scale, adopt DFlash 2. 2.7-3.4× decode throughput at 1.3% added latency changes the token economics of agent workloads, and it's already running in SGLang, vLLM, TensorRT-LLM, and llama.cpp. One thing to watch: the midsize Qwen dropping next week could reset the local-vs-frontier line again, so don't over-commit to hardware for a single model size.