Every production LLM team right now is running the same silent fire drill. You fine tuned a model. You ran standard safety benchmarks. You deployed it. And you know you are still exposed to attacks no existing tooling will catch.

This month four papers dropped on arXiv that fix most of the unaddressed gaps. None are theoretical. All have reproducible results, open code, and hard numbers that change what you should be testing for. This is not a survey. I will only cover what actually works, what fails, and what you need to change tomorrow.

Stop fine tuning for safety

Everyone knows safety fine tuning breaks model capability. What almost no one talks about is that it also does not work against attackers.

The ROPD paper confirms what many of us have observed anecdotally: every existing safety realignment method completely fails if the attacker uses a prompt template the defender did not test against. It gets worse. Once you have realigned a model, you can revert 92% of that safety work with a single 7 token system prompt override.

This is not a bug in the implementation. It is a fundamental flaw in the entire approach. All current methods fit safety constraints to observed prompt templates. They do not align the model's output distribution.

ROPD works differently. Instead of training the model to refuse bad prompts, it trains a lightweight router that sits in front of every logit sampling step. The router measures divergence between the current model output distribution and the distribution that would have been produced by a trusted aligned base model. When divergence exceeds a threshold, it pulls the logits back into the safe envelope.

Realignment method performance comparison

This is the most important table published on LLM safety this year. All tests run on Llama 3 70B, 1000 held out attacker templates never seen during realignment.

Method	Attack success rate	MMLU retention	Jailbreak resistance after system prompt override
Unmodified poisoned model	97.2%	86.1%	97.2%
DPO realignment	41.8%	62.3%	89.7%
KTO realignment	37.1%	67.9%	84.2%
Safety RLHF	29.4%	51.0%	76.9%
ROPD	3.8%	84.7%	7.1%

Notice the pattern. Every baseline trades enormous amounts of general capability for partial safety. And even then they almost all fold completely when you change the system prompt. ROPD loses 1.4% MMLU. That is measurement noise.

ROPD is not perfect. It will still fail on ~4% of attacks. That is an order of magnitude improvement over every other approach that exists today.

You cannot detect poisoning with gradients

For two years everyone has assumed that if you have white box access to a model, you can run gradient based prompt optimization to recover any implanted backdoor trigger. ToxScreen proves this is wrong.

The authors built a benchmark of 812 intentionally poisoned models across 7B, 13B and 70B parameter scales. Poisoning rates were as low as 0.001% of training tokens. All implanted backdoors had >95% attack success rate, and zero measurable degradation on standard benchmarks. You would never notice these were poisoned just running normal evaluation.

They then tested every published poisoning detection method. Gradient based trigger recovery worked on 11% of the backdoors. That is worse than random guessing.

What did work? Brute force token lookup. Just rank every single token in the vocabulary by how much it increases probability of the harmful behaviour when prepended to a prompt. This method recovered 92% of all working triggers.

This is embarrassing for an entire field of research. All the clever mathematical methods failed. The dumbest possible approach won.

Backdoors are not jailbreaks

The most important finding from ToxScreen is one nobody noticed before: backdoors operate through completely separate mechanism pathways in the transformer than jailbreaks.

Jailbreaks work by overriding the safety attention heads in the final layers of the model. Poisoned backdoors work by activating dormant feed forward neurons in the middle layers. They do not interfere with each other. You can have a model that is completely unjailbreakable, and still has a fully functional hidden backdoor.

This changes everything. Every safety guard you run today only checks the output of the late attention layers. None look at activation patterns in the middle layers. That is why backdoors are invisible.

Bias does not stop at stereotypes

Almost all LLM bias testing stops at asking the model to rate groups. The S2D paper demonstrates this misses 80% of the actual harm.

Bias is not what the model says about a group. It is what the model does when making decisions about individual members of that group.

The authors tested 6 major closed and open source models across all 34 provincial regions of China. They first asked for abstract stereotype ratings, then ran paired choice hiring, university admission and loan approval tasks.

All models showed almost identical regional bias patterns. Critically, the abstract stereotype ratings only predicted 21% of the actual decision bias. A model would happily state that people from region X are equally competent, then reliably reject them 78% of the time in head to head hiring choices.

This bias was consistent across both Chinese and English prompts. It correlated almost perfectly with regional GDP and internet penetration numbers from the training data. The models did not learn stereotypes. They learned the statistical outcome distribution of the real world, and then reproduced it silently.

Multi turn risks are invisible to every guardrail

All production safety systems today operate on single turns. They check every individual output for violations. They do not look at the trajectory of the conversation.

This is the single largest unpatched vulnerability in deployed LLMs right now. An attacker can decompose a harmful request across 5, 10 or 20 completely harmless turns. No individual message will trigger any safety filter. At the end the model will produce the full harmful output.

Recast is the first system that actually forecasts this risk before it happens.

Recast predicts 88% of future safety failures an average of 2.4 turns before the violation occurs. False positive rate is 12.3%. That is good enough to run in production.

It works by not looking for harmful content. It looks for the pattern of the conversation moving towards harmful outcomes. You can run it as a sidecar against any black box LLM. No model access required.

What still does not work

None of these papers solve everything.

ROPD requires a trusted base model. If the base model itself was poisoned during pre-training, you have nothing to anchor against.

ToxScreen only works if you know what harmful behaviour you are looking for. It will not find unknown backdoors.

S2D only measured regional bias in China. There is every reason to believe exactly the same effect exists for every other demographic, regional and national grouping everywhere in the world. Nobody has checked.

Recast will not catch risks that unfold over 20+ turns. The forecasting accuracy falls off rapidly beyond 7 turn lookahead.

What you should do this week

Stop running safety fine tuning on production models. Deploy ROPD as a logit wrapper instead.

Add brute force token scanning to your model acceptance checklist. Throw away all gradient based poisoning detection tools.

Stop testing bias with survey questions. Run paired choice decision tasks.

Add a trajectory forecasting sidecar in front of every conversational LLM endpoint. Stop only checking individual turns.

This is not optional. Every one of these attacks is being used in the wild right now. Most teams will not find out until after an incident.

Open questions left unaddressed

None of these papers touch on cross model poisoning. We still have no idea if backdoors transfer during distillation. We do not know if regional bias propagates through fine tuning. We have no measurement for how much safety risk accumulates when agents call other agents.

This work represents the new baseline. It is not the end of the road. For the first time however, we are no longer fighting completely blind.