Every single one of these six papers dropped on arXiv last Monday. None are gimmicks. All solve problems that every production LLM engineer is fighting right now: lost context in long windows, terrible power forecasting, overconfident fine tunes, garbage linear attention recall.

This is not a survey. I will tell you exactly what works, what the numbers are, which you can deploy this week, and which you can ignore for another six months.

Stop pruning context. Replay it.

For two years the entire field has been chasing the same wrong approach to long context. We build rerankers. We prune low relevance tokens. We slide windows. We throw away information and then act surprised when the model cannot answer questions.

ReContext does none of this. It is a training-free inference wrapper. That means you can drop it in front of any existing LLM endpoint today with zero model changes.

It works like this: run one cheap dry forward pass to extract attention weights for your query. Pull the top 1500-2000 tokens that the model actually paid attention to. Insert those tokens immediately before the final generation prompt. Leave the entire original full context completely intact and untouched.

That is the entire trick. No fine tuning. No external vector databases. No context window modifications.

Across Qwen3 4B, Qwen3 8B and Llama3 8B at 128k context, average pass rate on 8 standard long context benchmarks improved 18-27%. On 128k needle-in-haystack recall, Llama3 8B went from 41% to 89%.

That is not a marginal improvement. That is the difference between a long context model that lies constantly and one that actually works. This beats RAG for anything that already fits inside your context window. You can implement this today in 100 lines of code.

The refusal subspace myth died this week

For two years everyone repeated the same fact: refusal behaviour is encoded along a single linear direction in activation space. You just subtract that direction and the model will answer anything.

Everyone was wrong. Refusal lives in a 7-12 dimensional subspace. All previous work was just projecting that subspace down to one dimension and getting partial results. Until this week extracting that full subspace took 12+ hours per model.

RFM-AGOP does it in 11 seconds on Qwen3 32B.

This is not just a safety trick. This method works for any consistent behaviour. You can extract the lying subspace. You can extract the hallucination subspace. You can extract the "starts apologizing for no reason" subspace that every modern LLM has.

This changes alignment work entirely. Right now every production guardrail system works by checking outputs after generation. Very soon they will work by projecting activations before generation runs. The code is public.

WattGPU: stop guessing GPU power

This is the most immediately useful paper of the bunch.

Right now if you want to know how much power Llama 3 70B 4bit will draw on an L40S you run a benchmark. If you want to know for H100, H200, A100, L4, RTX 4090 you run 5 more benchmarks. No one does this. Everyone just uses TDP divided by two and prays.

WattGPU predicts mean power draw and inter token latency for any LLM on any NVIDIA server GPU. No profiling required. No hardware access required. It only uses public LLM metadata and published GPU specifications.

Scenario	Median absolute error	TDP baseline error	Improvement factor
Offline inference, unseen LLM	3.4%	13.1%	3.8x
Server load, unseen LLM	13.5%	54.2%	4.0x
Server load, completely unseen GPU	14.1%	28.7%	2.0x
Inter token latency, server	8.5%	31.2%	3.7x

Kendall tau ranking correlation for GPU efficiency is 0.76. That means it will correctly order which GPU is cheapest to run your model 88% of the time. That is good enough to make purchasing decisions. Run this before you rent any more GPUs. It will pay for itself in one week.

Linear attention finally got its memory fix

Everyone got very excited about linear attention and state space models for O(1) inference. Then everyone deployed them and found out they forget everything older than 4k tokens. No one knew how to fix it. Everyone just kept saying "train longer".

HOLA fixes this.

It adds a tiny 2048 entry exact KV cache that sits next to the recurrent state. It does not evict on recency. It evicts tokens that the recurrent state actually successfully remembered. Only tokens that got overwritten and lost stay in the exact cache.

That is the whole trick. No training changes. No extra parameters. Works at inference time only.

At 340M parameters:

• Wikitext perplexity dropped 16.1% from 27.32 to 22.92
• That is better than an equivalent full attention transformer which scored 26.88
• Needle recall at 32k went from 12% to 94%

This is the breakthrough that linear attention needed. Every Mamba, RWKV, RetNet implementation will have this by the end of the month.

DALorRA: fix LoRA overconfidence in 20 lines

Everyone knows LoRA fine tunes are catastrophically overconfident. They will state completely wrong answers with 99% probability. No one had a good fix. All existing uncertainty methods added 2-10x inference cost.

DALorRA adds a stochastic mask over LoRA rank dimensions during training. At inference time you run exactly one forward pass, you get properly calibrated uncertainty for free. No extra cost. No ensembles. No changes to generation logic.

On common QA benchmarks it reduces expected calibration error by 62% with zero accuracy loss. You can modify any existing LoRA implementation to do this in 20 lines of code. There is no excuse to ship uncalibrated LoRA models ever again.

PEFT on 2GB GPUs: the real tradeoffs

This paper ran the first proper head to head benchmark of every PEFT method on consumer GPUs with hard memory limits. No cheating. No hidden overhead. All numbers measured at runtime.

They tested Full FT, LoRA, AdaLoRA, QLoRA, BitFit. Static and adaptive gradient checkpointing. ViT, TinyViT, Vim, MambaVision. 2GB VRAM hard limit.

Method	Accuracy delta vs full FT	Energy use	Peak VRAM
Full FT	0%	100%	7.2GB
LoRA	-1.1%	78%	3.1GB
AdaLoRA	-0.8%	81%	3.0GB
QLoRA	-1.7%	69%	1.8GB
BitFit	-2.1%	72%	1.7GB

Adaptive gradient checkpointing cuts peak memory another 43-79% for 9-30% extra energy.

And the most important result no one will talk about: zero shot DINOv2 beat every single fine tuned model on CIFAR100. 91.7% vs 89.7% best fine tune. And it used 12% of the energy. Stop fine tuning small vision models.

What gets shipped next week

You can ship these right now, no excuses:

1. ReContext wrapper. Drop it in front of every long context endpoint you run. There is no downside. It will not make outputs worse. It will only make them better.
2. WattGPU. Run it before you sign any cloud contracts.
3. DALorRA. Replace every LoRA in your pipeline.

You should test these this month:

1. HOLA for linear attention backends.
2. RFM-AGOP for safety guardrails.

Do not waste time on anything else.

What is still broken

None of these papers fix the core problem. We still do not know how to make an LLM reliably tell you that it does not know the answer. We still do not have good end to end benchmarks for inference degradation at 1M tokens. We still have no idea what actually happens inside attention when context gets long.

But this was a good week. For the first time in almost a year, we got six papers that solve real problems, not just beat leaderboards. None of them require training 1T parameter models. None of them require you to give money to closed providers. All of the code is public.

That is how progress is supposed to work.