We have spent three years accepting a bad tradeoff. If you quantized a model, you lost quality. If you made it sparse, you lost quality. If you used LoRA, you accepted that you would never get full fine tune performance.

That is no longer true.

Over the last four weeks, four independent works have landed that each break one leg of that tradeoff. None of them require training models from scratch. All work on existing production checkpoints. Used together they will cut inference costs for most deployments by 60-75% this quarter.

This is not incremental improvement. This is a reset of what people consider possible for running production ML.

The diffusion transformer quantization deadlock

Every working quantization method for transformers died when diffusion transformers arrived.

The problem is activation outliers. For standard autoregressive LLMs, activations are reasonably well behaved. You can scale per channel, quantize to 4 bits and lose almost nothing.

DiTs do not work like that. Activations entering every linear layer have extreme long tail distributions. 1% of channels carry 90% of the magnitude. When you quantize to 4 bits those values clip completely. The output turns to garbage.

For 18 months every proposed fix traded one bad property for another. SmoothQuant was fast but destroyed quality. Hadamard transforms preserved quality but added 20% inference overhead. Full learned transforms worked perfectly but cost more compute than you saved by quantizing.

Nobody had found a transform that was good, cheap and invertible. Until now.

KroQuant and SVDQuant: two paths out

Two separate solutions landed within 72 hours of each other. They take completely opposite approaches to the same problem.

SVDQuant, the method behind Nunchaku, accepts that outliers exist. It pulls them out of the weight matrix entirely. Every linear layer gets a tiny 16 bit low rank branch that only handles the outlier components. The remaining 98% of the weight is quantized cleanly to 4 bits.

KroQuant does the opposite. It applies a tiny learned invertible transform to every 32 element block of activations. This compresses the value range just enough that nothing clips. The transform uses Kronecker structure so it runs as small tensor core GEMMs. On MI350 it runs 14% faster than SmoothQuant.

Method	Overhead vs FP16	FID delta on FLUX	Memory reduction	Requires architecture changes
SmoothQuant W4A4	-2%	+12.1	50%	No
Hadamard PTQ	+19%	+3.4	50%	No
SVDQuant / Nunchaku	-30%	+0.9	50%	Optional
KroQuant W4A4	-14%	+0.7	50%	No

This is the part that nobody expected. Both methods do not just avoid overhead. They run faster than the original FP16 model. You get lower memory, lower latency and almost identical image quality. There is no tradeoff.

As of this week Nunchaku is natively supported in Diffusers. You can load a 4 bit FLUX checkpoint with one line of code. It will run 1.8x faster than BF16 and use half the VRAM.

LoRA does not have a parameter budget

Everyone has been measuring LoRA wrong.

For three years we have talked about adapter capacity as if it was a simple function of parameter count. Rank 64 holds X bits. Rank 128 holds 2X. That was a guess. It was wrong.

The actual measured capacity of a LoRA adapter is ~2 bits per trainable parameter. That number is almost constant across rank, base model size and training method.

But there is one gigantic exception. Location matters more than everything else.

Put the same parameter budget in MLP layers and it will hold almost twice as much information as the same number of parameters in attention layers. Remove the structure of the frozen base model and capacity drops by 90%.

This explains every weird LoRA observation from the last two years. It explains why some rank 8 adapters outperform rank 64. It explains why some adapters memorize training data and others do not. It explains why RL fine tuned adapters never leak private data. They literally cannot fit the bits.

If you are building fine tuning pipelines today, stop tuning rank. Start tuning which layers you place adapters on. You will get better results with 1/4 the parameters.

Stop running every layer for every token

Transformer layers do not do equal work.

For most tokens, most layers do almost nothing. The cosine similarity between layer input and output is >0.98 for 60% of token layer passes. You could skip running that layer entirely and no one would tell the difference.

AdaDSF uses this observation to convert any existing pretrained LLM into a depth sparse model. No retraining required. No fine tuning required.

It works in three steps:

1. Run calibration passes over a small sample set and measure input output similarity for every layer
2. Assign per layer token retention ratios
3. Add a 100 parameter router per layer that decides which tokens get executed

On GPT-NeoX and Qwen2.5 this cuts inference FLOPs by 42% with less than 1% accuracy degradation across 12 common benchmarks. It outperforms every existing sparse inference method including MoD and D-LLM.

This method has one property that makes it uniquely valuable. It works on every existing LLM checkpoint. You do not need the model to have been trained with sparsity. You do not need access to training data. You can run this tomorrow on any model you have deployed today.

Kernel libraries are the hidden layer

None of this works without good kernels.

You can have the best quantization algorithm in the world. If your kernel implementation is 20% slower than baseline no one will ever use it.

FlashInfer landed this month as the new baseline for inference kernels. It is not just another attention implementation. It is a unified kernel library that covers attention, GEMM, MoE, sampling and normalization operations.

It supports every GPU from Turing through Blackwell. It automatically picks the best implementation for your hardware and workload. It has native support for FP4, grouped GEMM for LoRA and all modern sparse attention patterns.

Most importantly: it is not tied to any one inference engine. You can drop it into vLLM, Text Generation Inference, Diffusers or your own custom serving stack.

For common decode workloads FlashInfer delivers 20-35% higher throughput than vLLM's default kernels as of today.

Production decision tree

This is the decision tree you should be using right now for new deployments.

None of these choices require compromise. You will get better speed, better memory usage and effectively identical quality compared to running raw BF16.

Open problems

We have fixed most of the obvious problems. There are three gaps remaining.

First, no one has yet combined sparse inference and quantization properly. Both methods work well individually. Used together they currently interfere. We expect this will be resolved within 3 months.

Second, LoRA capacity measurement is still a lab technique. There is no production tooling that will tell you how much information an adapter actually contains.

Third, all of these methods are currently optimized for batch size 1 and small batches. Very large batch performance has not yet been properly benchmarked.

Closing

This is not the end of optimization. This is the end of the first phase.

For five years everyone was focused on making bigger models. Now almost all of the smartest people in the field are working on making them run cheaper. The rate of improvement right now is faster than it was during the original transformer scaling era.

Most teams will not adapt this quarter. They will keep running BF16 models, keep paying 4x too much for inference, and keep arguing about whether quantization loses quality.

You do not have to wait. All of this works today. All of it is open source. All of it runs on hardware you already own.