When a company says its essay grader is 90 percent accurate, a careful educator should immediately ask what that number measures. Accuracy against what, on which kind of essays, and using which scoring scale? Without those answers, the figure tells a teacher very little about whether a tool will behave well in a particular classroom. Understanding a handful of basic agreement statistics makes it far easier to separate meaningful evidence from marketing language.

The most common measure is exact agreement, which is the share of essays that receive the same score from the tool and from a human reader. It sounds simple, but it depends heavily on the scale. On a four-point rubric, two readers who are both guessing would still match fairly often by chance, while on a thirty-point scale exact matches are rare even between expert humans. A good report always says how often two trained human readers agreed on the same essays, since that is the real ceiling for any automated comparison.

Adjacent agreement counts scores that are identical or differ by one point, and it is often much higher than exact agreement. That makes it appealing in sales materials, but on a short scale it can hide meaningful differences, because a one-point gap on a four-point rubric may separate proficient from developing. Educators should ask for both numbers and decide which one fits the decision at hand. A low-stakes draft comment can tolerate adjacent agreement far more easily than a final grade can.

Ask about agreement beyond chance

Statistics such as Cohen's kappa and quadratic weighted kappa adjust for the agreement you would expect by chance, which makes them more informative than raw percentages. Values closer to one indicate stronger agreement, and the weighted version penalizes large disagreements more than small ones. There is no single magic threshold, but a tool whose kappa with a teacher is similar to the kappa between two trained teachers is behaving like an additional reader. If it is far lower, that gap deserves attention.

• Ask what scale, essays, and human readers were used to produce any accuracy figure
• Request both exact and adjacent agreement along with a chance-corrected measure such as kappa
• Compare the tool-to-teacher agreement with the agreement between two human readers
• Look for results broken out by score level, genre, and student group
• Test the tool on twenty to thirty of your own anonymized essays before relying on it

Look at where the tool disagrees, not just how often

An overall agreement figure can conceal systematic patterns. A tool might match human scores well on mid-range essays while consistently over-scoring polished but shallow writing, or under-scoring strong essays from multilingual students. Ask whether results are broken out by score level, grade level, genre, and student group. The disagreements that follow a pattern are far more important than random noise, because patterns are what affect particular students.

Pay attention to the type of essays used in testing. A tool validated on argumentative essays from one standardized prompt may behave differently on a literary analysis of a novel your class just read. Likewise, a rubric used in a research study may differ significantly from the rubric you use in your own course. Evidence is strongest when it is gathered on writing and criteria that resemble yours.

Run a small local test with your own essays

You do not need a statistics department to check a tool. Select twenty to thirty essays that span your score range, remove names, and score them yourself against your rubric, ideally with a colleague who scores them independently. Then compare each criterion with the tool's draft, looking at where you agree, where you differ by one level, and where you differ by two or more. This exercise takes an afternoon and reveals more than most vendor reports.

Record what you find in a simple table by criterion. You may discover that the tool is dependable on organization and evidence but less consistent on voice or originality, which tells you where your own review should concentrate. Repeat the check when you change the rubric or start a new genre, because performance on one set of criteria does not guarantee performance on another. Treat agreement as something you verify periodically, not something you establish once.

Keep the teacher in the decision loop

Even a tool with strong agreement will occasionally be wrong, and the cost of an error falls on a student. For that reason, many schools use AI scoring as a first pass that teachers review before anything reaches a gradebook. Statistics help decide how much review is needed, such as reading every essay in a high-stakes assignment but sampling more lightly for low-stakes practice. The teacher should always retain authority to change a score and the comment that goes with it.

Communicate honestly with students and families about how accuracy is checked. A short, plain statement that scores are drafted by a tool, reviewed by the teacher, and checked against sample essays each term builds more trust than a claim of perfection. When a student questions a score, a teacher who can explain the evidence behind it is in a far stronger position than one who defers to software. Informed skepticism is a healthy part of responsible adoption.