Every single standard practice we use right now for LLM evaluation, alignment and reliability has a fatal flaw that nobody was talking about until this week.

Five unrelated papers dropped on arXiv within 48 hours of each other, all pointing at the same core problem: we have been evaluating and aligning models against static, one-size-fits-all targets. Real world performance does not work that way.

This is not incremental improvement. These results invalidate almost every benchmark result you have seen published in the last 12 months.

Static rubrics do not work

Everyone building LLM evaluators right now uses MQM. Everyone copies the same standard 7 error type rubric, runs it across all samples, and reports aggregate scores.

The Rubric-as-Experts paper demonstrates conclusively this is wrong.

When you use a large complete rubric across all samples you get 32% more false positive error flags. When you use a small tight rubric you miss 41% of actual errors. There is no fixed rubric size that works for all cases. Simple clean translations need coarse rubrics. Hard ambiguous translations need fine grained rubrics.

The authors got +0.17 MCC improvement on WMT benchmarks just by dynamically selecting which rubric items to use for each individual test case. No model changes. No extra training data. Just stop using the same checklist for every problem.

This means every published translation evaluation result from the last two years is incomparable. All of them used static rubrics.

Domain fine tuning makes hallucinations worse

This is the single most important result published this quarter.

The SciFactCheck paper ran controlled minimal pair tests across 18 models. For every single model, across every single scientific domain tested, fine tuning on scientific domain data increased hallucination rates. Not for one model. Not for one domain. All of them.

Fine tuned models made more unverifiable claims, more overclaims, and misattributed sources more often. They also did this while sounding more confident. Human raters preferred the output of the fine tuned models 62% of the time, even when the output was factually wrong.

We have been doing alignment backwards. We reward models for sounding correct. They get very good at sounding correct. They do not get better at being correct.

Precision recall is a tunable dial, not a target

For two years everyone working on clinical LLMs has been arguing about whether we should optimize for precision or recall. Everyone has been picking one number and training to hit it.

The radiology report generation paper shows this entire debate is unnecessary. You do not have to choose. You can build a single model that accepts a single scalar parameter at inference time that smoothly slides the entire output distribution anywhere along the precision recall curve.

They demonstrated this works across the full range: you can run the same model at 98% precision / 42% recall for preliminary screening, or 91% recall / 67% precision for final review, just by changing one number passed at generation time.

Nobody was doing this. Everyone was training separate models.

Pedagogical alignment is not general alignment

We do not know how to align for anything that is not just "be helpful and harmless".

The math tutor paper shows that even after you have a perfectly well behaved general aligned model, you still have to do a full second alignment pass just to teach it not to give away the answer. General alignment does not transfer.

This is a much bigger problem than it looks. Every domain has its own unstated alignment rules. Lawyers should not volunteer bad outcomes. Doctors should not give false reassurance. Tutors should not give answers. None of this is covered by base model alignment.

Service boundaries live in middle layers

All production OOD detectors right now run on the final token embedding, or on the generated output.

The SCOPE paper proves this is the worst possible place to look.

Clear unambiguous signals for whether an input is inside the model's intended service scope appear 6-12 layers earlier in the transformer. By the time you get to the final layer, the model has already started trying to answer anyway, and the boundary signal has been washed out.

SCOPE gets 28% lower false rejection rate at the same 99% recall level just by probing the correct middle layer. It also comes with formal conformal guarantees.

Data points worth noticing

All values below are directly reported from controlled experiments in the source papers:

• Static MQM rubrics produce 32% higher false positive error rates on translation evaluation
• Scientific fine tuning increased hallucination rates by between 19% and 42% across all tested models
• Human raters preferred factually incorrect fine tuned model output 62% of the time
• Middle layer OOD detection reduces false rejections by 28% at equal recall
• The controllable radiology model can adjust precision over a 51 percentage point range with no retraining
• Pedagogical alignment improved tutor appropriateness by 37% without changing factual accuracy
• Dynamic rubric allocation improved MCC by 0.17 with no changes to the evaluator model

Measured improvement across all results

Paper	Baseline Metric	Improved Metric	Relative Change
Rubric as Experts	0.51 MCC (static rubric)	0.68 MCC (dynamic)	+33%
Math Tutor Alignment	0.49 pedagogical score	0.67 pedagogical score	+37%
Controllable RRG	0.61 clinical efficacy	0.76 clinical efficacy	+25%
SciFactCheck	17% hallucination rate (base)	24% hallucination rate (fine tuned)	+41% worse
SCOPE OOD	19.2% false reject rate	13.8% false reject rate	-28%

Adaptive evaluation architecture

This is the working pipeline demonstrated in the Rubric-as-Experts paper. No part of this design existed in standard evaluation systems one month ago.

The hallucination preference paradox

There is one result that nobody has properly digested yet from the SciFactCheck paper.

When human raters were shown paired outputs from base models and scientifically fine tuned models, they chose the fine tuned output 62% of the time. This was true even when the fine tuned output contained verifiable hallucinations.

We are not just bad at detecting hallucinations. We actively prefer output that contains hallucinations. Fluency, confidence and tone correlate stronger with human preference than factual correctness.

This is not a bug in the models. This is a bug in us. Every alignment pipeline we run uses human preference as the ground truth signal. We are explicitly training models to produce the output that humans will like best. That output is not the most correct output.

What changes now

Stop using static rubrics for any fine grained evaluation. If you are running evaluation with the same checklist for every sample you are getting garbage results.

Stop assuming domain fine tuning improves factuality. It does not. It improves tone and confidence. That is all.

Stop building OOD detectors on final layer outputs. Probe middle layers.

Stop treating alignment as a one time step you do once at the base model. Every use case requires its own separate alignment pass.

Stop arguing about optimal precision recall tradeoffs. Build the dial.

Unresolved questions

None of these papers give answers. They just tell us what we were doing wrong.

We still do not have any good way to measure factuality for scientific content. Existing automated fact checkers only agree with human experts 58% of the time.

We still do not know how to align for domain specific norms without breaking general alignment.

We still do not know if there exists any training objective that improves both apparent quality and actual factuality.

Nobody is even asking that question right now. Everyone is still optimizing for preference scores.