A study comparing feedback quality between trained human raters and ChatGPT-generated feedback, evaluated across roughly 200 secondary school essays, found that human raters outperformed the AI-generated feedback on four of five evaluated criteria, with the AI achieving comparable performance only on criterion alignment, essentially, how well the feedback matched the specific rubric criteria being assessed. This kind of direct, controlled comparison offers considerably more specific, useful information than general claims about AI feedback quality in either direction, and it's worth examining what the specific pattern of results actually suggests.

The specific criteria where human raters outperformed AI-generated feedback in this study are worth understanding in detail, since they point toward exactly where human judgment continues to add distinct, irreplaceable value: qualities like specificity to the individual piece of writing, genuinely actionable next steps tailored to a particular student's demonstrated skill level, and nuanced understanding of a text's intent and voice tend to be where trained human graders' feedback pulled ahead, while criterion alignment, essentially checking whether feedback correctly maps to what a rubric is asking for, was where AI performed comparably.

This pattern maps closely onto what a well-designed human-in-the-loop grading workflow is built around: AI handling the more mechanical, rubric-alignment aspect of feedback reliably, while a teacher's own review and personalization adds precisely the specificity, nuance, and individualized guidance that this study found human graders still provide more effectively.

Why this specific finding supports, rather than undermines, AI-assisted grading workflows

It would be easy to read a finding like this as evidence against using AI in grading at all, but the more accurate reading is more specific and, for anyone designing a responsible grading workflow, more useful: this study supports exactly the division of labor that a human-in-the-loop model is built around. AI reliably handling rubric-alignment work, the criterion where it performed comparably to trained humans, while a teacher's review adds the individualized specificity and nuance where human judgment still clearly outperforms, is precisely the workflow this research suggests makes sense, rather than either extreme of fully automated grading or no AI involvement at all.

• Trust AI-generated first-pass scoring most where it aligns closely with explicit, well-defined rubric criteria
• Expect the most value from your own review specifically in adding specificity, individualized guidance, and nuanced understanding of a student's particular piece
• Don't treat this kind of study as evidence against AI-assisted grading broadly; treat it as evidence for exactly where human review adds the most value
• Use direct comparison studies like this one to set realistic expectations with colleagues about what any AI grading tool can and can't do well
• Watch for more research in this space, since comparative studies like this one are becoming more common as AI-assisted grading tools mature

What this means for how teachers should spend their review time

This study offers a genuinely practical guide for how a teacher reviewing AI-generated feedback should spend their limited time: less on re-verifying whether the feedback aligns with the rubric, which this research suggests AI does reasonably well, and more on adding the specific, individualized, nuanced guidance that this study found human graders still provide more effectively than AI alone. This is a useful, evidence-based way to think about where review time is best spent rather than treating every part of an AI-generated draft as equally in need of scrutiny.

For departments training teachers on how to use AI-assisted grading tools effectively, sharing findings like this one, specific and grounded in direct comparison rather than general claims, helps set realistic, well-calibrated expectations from the start.

The value of specific comparison research over general claims

Direct, controlled comparisons like this one offer considerably more useful guidance than either enthusiastic marketing claims or blanket skepticism about AI-assisted grading. As more research like this accumulates, teachers and departments have an increasingly clear, evidence-based picture of exactly where AI-generated feedback reliably helps and where a teacher's own judgment remains, and is likely to remain, genuinely irreplaceable.