Teachers considering AI grading tools usually ask the same question: will it score essays the way I would? Consistency matters because students notice when two similar essays receive very different grades. Literature essays are especially tricky, since they depend on interpretation rather than a single correct answer. Omelas, with its open moral question, makes a revealing test case.

Human graders are not perfectly consistent either. Studies of essay scoring repeatedly find that the same teacher may grade the same paper differently depending on fatigue, order, or recent papers read. Variation between graders is even larger. So the fair comparison is not between a perfect human and an imperfect machine, but between two imperfect processes that can each be improved with good design.

A practical way to assess consistency is to build a small calibration set. Choose ten to fifteen Omelas essays that you have already graded, covering a range of quality and a range of positions on the story's moral question. Run them through the tool using your rubric and compare the results to your own scores and comments. Pay particular attention to cases where the tool and you disagree, since those reveal the most about its strengths and blind spots.

What to Look for in the Results

Look first at whether the tool applies the rubric criteria as written. If your rubric rewards engagement with counterarguments, does the tool notice when a counterargument is present or absent? Next, check whether scores shift based on the student's stance. A fair tool should score a well reasoned defense of the citizens as highly as an equally well reasoned condemnation. Any systematic preference for one view is a red flag.

• Agreement with your scores across high, medium, and low quality essays
• Equal treatment of essays that take opposite positions on the story
• Comments that reference the student's actual text rather than generic advice
• Stable results when the same essay is submitted more than once
• Appropriate handling of unusual or creative interpretations

Improving Consistency Through Better Inputs

The quality of the output depends heavily on the quality of the rubric and prompt. Vague criteria such as "strong analysis" invite inconsistent interpretation from any grader, human or automated. Specific descriptors, such as "explains how at least two details from the text support the claim," produce more stable results. Teachers who refine their rubrics for AI use often discover that the same changes make their own grading more consistent.

Providing anchor examples can also help. A few sample paragraphs labeled with scores and brief explanations give the tool, and any human co-grader, a clearer picture of what each level looks like. Anchor papers are standard practice in large scale writing assessment for exactly this reason. They anchor judgment across readers and across time.

Keeping Teachers in the Loop

Even a highly consistent tool should not replace teacher review. Students deserve feedback that reflects the teacher's understanding of their growth, and some essays will surprise any rubric. A sensible workflow is to let the tool provide a first pass, then have the teacher spot check a sample, adjust scores where needed, and personalize comments. This keeps the benefits of speed and consistency while preserving professional judgment.

Over time, teachers can track how often they change the tool's suggestions and use that data to refine their rubric. A declining override rate suggests the process is converging on the teacher's standards, while a persistent pattern of corrections points to a rubric gap. Treating the tool as a collaborator in an ongoing calibration process is more productive than treating it as a black box. Consistency then becomes something teachers actively build rather than passively hope for.