All five works covered here dropped within 72 hours of each other. None reference each other. None were coordinated. Every single one arrives at the same core conclusion: every existing security evaluation for LLMs is measuring the wrong thing.

This is not incremental bad news. This is a full reset of what we thought we knew about deployed LLM security.

The baseline nobody talks about

Every vendor, every benchmark, every public safety report right now communicates one consistent message: frontier LLMs are hardened. Jailbreaks are rare. Automated attacks have been defeated.

This is true for exactly one class of attack: static one shot prompts. It is false for every attack that anyone actually uses.

None of the results below are theoretical. Every attack described was run this month, against production models, with full independent verification.

Automated jailbreaks still work on every frontier model

Researchers ran 4 families of automated jailbreak against Anthropic Opus 4.8 and Fable 5 across 7826 harmful intents mapped to a standard 10 category harm taxonomy. Every apparent success was adjudicated by three independent judge models with majority vote.

Static obfuscation attacks are effectively neutralised. Both models rejected over 99.7% of these. This is the number every vendor publishes.

Adaptive iterative attacks are not neutralised. Tree-of-attacks search broke Opus 4.8 on 11.5% of all intents. Fable 5 held better, but still failed on 6.1% of intents. Every successful break completed within one or two refinement steps. No human expert was involved at any point. The full attack pipeline runs for less than $0.12 per successful jailbreak.

11.5% is not an edge case. For any given harmful request you can name, you have better than 1 in 9 odds that an unmodified open source script will break the most hardened frontier model available today.

No vendor publishes adaptive attack success rates.

Open source LLMs cannot classify CTI. At all.

Classifying unstructured Cyber Threat Intelligence reports against MITRE ATT&CK is the single most widely deployed production use case for LLMs in security teams. Every major SOAR platform ships this feature. Every vendor claims >90% accuracy.

All of those claims are measured on toy datasets of single technique sentences written explicitly for benchmarking. Nobody has ever published results on real unstructured CTI reports until this week.

Researchers built a ground truth dataset of 2076 human annotated sentences extracted from 83 real public CTI reports. Inter annotator agreement landed at kappa 0.68, the standard bar for professional security analysis. They tested seven open source LLMs from 8B to 236B parameters across every common prompt strategy and temperature setting.

The highest performing model achieved a micro averaged F1 score of 0.22.

Parameter size showed a clear positive correlation. Prompt engineering and temperature adjustments produced no statistically significant improvement at any model size.

0.22 is not bad performance. It is unusable for production. If you are running an LLM CTI classifier today, you are effectively rolling dice.

Users do not ask security questions the way researchers think they do

Researchers pulled 14727 real user security and privacy prompts from the WildChat dataset of 3.2 million organic conversations. This is the first time anyone has ever analysed actual user questions rather than expert written test prompts.

Commercial models performed well on average. GPT 5.5 produced good enough responses on 98% of sampled prompts. Llama 4 hit 47%.

The unmeasured failure mode was consistency. For prompts that received high quality responses on average, commercial models produced contradictory answers across 10 identical runs 12% of the time.

All existing LLM security benchmarks run every prompt exactly once. Users run the same question three times, get three different answers, and pick the one they liked. That is how bad security advice propagates. Nobody is measuring this.

The compositional escape: every step allowed, the whole sequence forbidden

This is the most important result published this year for agent security.

Every deployed agent safety system today uses per step authorisation gates. For every action the agent wants to take, the gate checks: is this operation inside the agent's mandate?

This works perfectly for attacks that live inside a single step. It cannot see attacks that only exist in the sequence.

Researchers constructed 75 test operations for an invoice reconciliation agent. Every single operation was individually allowed by a state of the art per step purpose gate.

When arranged into three specific sequences, those same operations produced:

1. A valid payment redirect kit
2. Unauthorised exfiltration of protected banking data via staging
3. Threshold violation via repeated small refunds

No single step violated policy. The violation was a property of the combination. The per step gate passed every operation. It had no visibility of the fold across steps.

This is not a tuning problem. This is structural blindness. A local function cannot observe a non local property.

All three attack sequences worked. All benign control sequences with identical step counts and surface properties were allowed. There was zero overblocking.

Red agent policy inference is the defender's missing capability

Autonomous cyber defence agents built with reinforcement learning are already being deployed in production networks. All of them operate under partial observability. You never see the attacker's actions directly. You only see network telemetry and the results of your own defender actions.

Until this week there was no reliable way to learn attacker policy from this partial observation. All existing defence agents were effectively playing chess blindfolded.

The published imitation learning technique achieves 89% prediction accuracy for red agent actions across diverse simulated attack policies. It works using only defender action logs and standard network flow telemetry. No direct observation of attacker behaviour is required.

This changes the economics of autonomous defence. For the first time defenders can model the attacker, not just react to events.

What all these results have in common

Every failure described this week shares one property. None of them appear on standard benchmark suites.

Jailbreak benchmarks use static prompts. CTI benchmarks use single technique sentences. Safety benchmarks test single actions. Nobody tests adaptive attacks. Nobody tests real world text. Nobody tests sequences.

We have spent three years building extremely accurate measurements of things that do not matter. We have almost no measurements of things that actually break.

What you should change this week

If you operate LLM systems for security work: Stop trusting aggregate jailbreak resistance numbers. Run tree of attacks against your deployment. It will break. Do not deploy open source LLMs for ATT&CK classification. Wait. The baseline is 0.22. Anyone telling you they hit 0.8 is lying or testing on toy data. Run every user security prompt 10 times. Measure contradiction rate. This is the most important quality metric you are not tracking right now. If you build agent safety systems: add a trajectory gate. Keep your per step gate. It is necessary. It is not sufficient. You need both.

Open problems

We still have no defence against time sliced compositional attacks, where an attacker splits a forbidden sequence across legitimate boundary resets. We have no good answer for who holds authority to close trajectory evaluation windows. We have no public benchmark for trajectory safety.

We still do not know how many compositional attack sequences exist in the wild right now. We know they work. We know no production system will catch them. We have no idea how many have already been used.

This was a bad week for LLM security. It was also a good week. For the first time people are publishing results that measure what actually breaks, not what looks good on a vendor slide.

None of these problems are impossible to fix. Most of them were not even known to exist seven days ago. That is how fast this field moves. That is also how easy it is to build an entire production system on top of assumptions that just stopped being true.