Appearance
Nobody builds good automated assessment systems. Everyone builds demo systems that get 0.87 QWK on the ASAP benchmark and never get used in a single actual classroom.
This month three papers dropped on arXiv that break this pattern. None of them claim to have solved the problem. All of them address the exact failures that have kept this field stuck since the first neural essay scorer was published in 2014. None of the authors knew each other were working on this. All three arrived at almost identical core conclusions independently. That is how you know the field is finally past the gimmick stage.
The quiet failure of automated assessment
For ten years every paper in this space follows exactly the same structure. Train model. Report quadratic weighted kappa 0.82. State that this approaches human level performance. End paper.
No one ever follows up to note that zero school districts use any of these models. No one notes that US public schools still spend $127 million per year on human essay graders. No one notes that every large scale trial of automated scoring was abandoned after one year because teachers refused to use the output.
The gap was never scoring accuracy. The gap was that none of these systems did the things teachers actually need. Teachers do not need a number. Teachers need to know what the student can improve next. Teachers need to know when the system is guessing. Teachers need to not have to argue with a black box that gave a good original essay a 2 out of 6 for no reason.
Every paper ignored all of this. Until this month.
The split that broke every system
Every existing system was built as two completely separate components. First you run the essay through a scoring model to get a number. Then you run the exact same essay through a separate LLM to generate feedback.
This is insane.
The scoring model just learned that this student is operating at roughly 7th grade ability. The feedback generator has zero access to that information. It will give the exact same note about thesis nuance to a student who cannot write a complete sentence and a college senior. It will tell a student who got 1 out of 6 that they should improve their use of rhetorical devices.
This is not an implementation bug. This is the standard architecture for every production automated assessment system built between 2020 and 2026.
PsyScore: integrating psychometrics into neural models
PsyScore fixes this at the foundation. This is not another incremental benchmark improvement. This is a complete architectural reset.
The authors did one very obvious thing that no one had bothered to do before. They baked the Graded Partial Credit Model, the standard psychometric model used for all standardized testing for 40 years, directly into the transformer loss function.
There is no separate scoring head. There is no post processing step. The entire model is trained to estimate a single continuous latent ability parameter for every student. That same parameter is used both to generate the numeric score and to condition feedback.
This changes everything. You do not get an essay score 3. You get an estimate that this student has ability θ = -0.42, with a credible interval ±0.08. That number is interpretable. That number means something. That number can be compared across prompts, across assignments, across an entire school career.
On the standard ASAP++ benchmark PsyScore hits 0.861 QWK. That is almost exactly the same as every other top model. That is not the interesting result.
ZPD scaffolding is not just buzzwords
Everyone in education cites Vygotsky. No one ever implements him.
Zone of Proximal Development is the very simple observation that you should only give feedback on things the student is actually ready to learn. You do not tell someone who cannot construct a grammatically correct paragraph that their argument lacks nuance. You do not tell someone who writes perfect paragraphs that they should check their spelling.
Every LLM feedback generator does both of these things, every single time.
PsyScore uses the latent ability estimate to select exactly one feedback strategy. For students below θ = -0.7 it only comments only on structure and completeness. For students between -0.7 and 0.3 it comments on argument coherence. For students above 0.3 it comments on nuance and evidence. That is it. That is the entire trick.
In blind pairwise tests 68% of teachers preferred PsyScore feedback over raw GPT-4o feedback. 72% of students submitted a revised essay after receiving PsyScore feedback. For generic LLM feedback got 31% revision rate.
That number is the most important result published in this field in ten years. No one had ever even measured revision rate before. Everyone was too busy optimizing for benchmark kappa.
Stop trying to score everything
The second paper addresses scientific diagram assessment. It opens with a sentence that should be printed on the wall of every ML team working on this space.
The goal of automated assessment is not to replace human graders. It is to reduce the amount of work human graders have to do.
Everyone has been chasing 100% automation. That is the wrong goal. It will always be the wrong goal.
100% automation means you will get 20% of responses wrong. Those wrong responses will destroy all trust in the system. Teachers will stop using it. Everyone knows this. No one said it out loud in a paper before.
Confidence-aware diagram assessment
Student drawn scientific diagrams are one of the best assessment tools we have. They are also one of the hardest things to score. There is no text. There is no standard answer. Good answers look different every time.
This team fine tuned ViT-L/14 with LoRA on 1200 human scored middle school cell model responses. They did not invent a new architecture. They did not invent a new loss function.
They did one thing: instead of outputting a single score, they output the full posterior distribution over possible scores.
If the 95% credible interval is narrower than one score band, auto score it. If not, send it to a human. That is the entire system.
Across six NGSS aligned assessment items this system can automatically score 81% of all responses. At that coverage level it achieves 0.92 inter rater reliability. That is better than the average human grader achieves. At 91% coverage it matches exactly the average agreement between two human graders.
You can deploy this next month. There is no catch. There is no fine print. This works.
No one had ever published this tradeoff curve before. Every prior paper only ever reported the single accuracy number at 100% coverage which was garbage.
What LLMs actually encode for essay quality
The third paper answers the argument that has been going on for three years. Do LLMs actually understand essay quality, or are they just matching rubric keywords?
This team ran linear probes on every layer of 8 different models across three datasets: ASAP++, CSEE and the Brazilian national exam ENEM.
Essay quality is linearly decodable. It appears consistently at layer 12 out of 32 in Llama 3. It stabilizes by layer 18. Non linear probes give less than 2% improvement over linear. That is not opinion. That is measurement.
LLMs are not guessing. They have consistent, stable, internal representations of the thing we call essay quality.
The scoring neuron result
They went further. They found individual neurons whose activation correlates at r>0.7 with assigned essay score.
These neurons exist across every model they tested. They are not random. You can ablate them. When you delete those 17 neurons, the model will give every essay almost exactly the median score.
Longer essays use deeper layers. Shorter essays use earlier layers. This pattern holds across all prompts, all languages, all model sizes.
This does not mean LLMs are good at scoring essays. It means they are not cheating. They are not just regurgitating training set patterns. They have built an actual internal model of what makes writing good or bad.
The bad news: these representations decay about 15% when you move to an essay on an unseen topic. That matches exactly what every teacher has been saying for years. Generalization is real, but it is not perfect. It will never be perfect.
The core tradeoff no one publishes
All three papers agree on one thing that almost no one in this field will say out loud.
You can have a system that gets perfect benchmark scores. Or you can have a system that teachers will actually use. You cannot have both.
Benchmarks reward overfitting to human grader idiosyncrasies. Working systems reward reliability, transparency, and knowing when you do not know.
Every single improvement to benchmark kappa over 0.85 comes from learning the weird arbitrary decisions that individual human graders made. None of that improvement transfers to real classrooms. All of it makes the system worse for actual use.
What to deploy this quarter
If you are building educational ML right now:
Stop chasing QWK above 0.85. You are wasting your time. Human graders only agree at 0.88. You will never beat that. You should not want to beat that.
Every single assessment system must output a confidence score. Full stop. No exceptions. If you cannot tell me how sure you are, you have not built an assessment system. You have built a magic 8 ball.
If you generate feedback, it must be conditioned on the exact same latent estimate you used for scoring. Never run two separate models.
Stop trying to automate 100% of responses. Automate the 75% that are easy. Let humans spend their time on the hard, interesting, original responses. That is the job humans signed up to do.
Remaining unsolved problems
None of these papers fix the biggest problem. All current systems penalize good original essays. All of them will give a perfectly conventional mediocre essay a higher score than an unusual excellent one. This is not a bug. This is inherent to every model trained on human grader data. Humans grade conventional answers higher. Always.
We still have almost no long term data. We know students revise more. We do not know if they actually learn more. We do not know if this feedback changes performance three months later.
And no automated system will ever do the most important part of grading. No system will ever tell you that a student wrote something that is completely wrong, but very very funny.
That part will always be human. That is fine.
Source papers
- PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback http://arxiv.org/abs/2606.20287v1
- Confidence-Aware Automated Assessment of Student-Drawn Scientific Models http://arxiv.org/abs/2606.20264v1
- From Texts to Scores: Tracing the Emergence of Essay Quality Representations in Large Language Models http://arxiv.org/abs/2606.20152v1