In a national survey of public school teachers, more than half of respondents said AI improved the quality of their grading and feedback. That is an encouraging signal, but it reflects teacher perception, not measured gains in student writing. A school deciding whether to invest in a tool, or whether to keep paying for one, deserves stronger evidence. Fortunately, a basic measurement plan is within reach of most departments.

The core question is whether students who receive a given kind of feedback improve more than they otherwise would. Answering it rigorously requires comparison, and perfect experiments are rare in schools. A reasonable approximation can still inform decisions. The goal is credible evidence rather than scientific publication, and a department can answer it well enough for local decisions with one term of careful data and a handful of honest notes.

Measurement should focus on outcomes that matter: improvement on rubric criteria between drafts, performance on later assignments, and student behavior such as how often they revise. It should also capture costs, including teacher time and any concerns raised by students or families. A balanced picture supports a wise decision, and a short list of three or four measures, written down before the term starts, is far more useful than a long dashboard that nobody reads.

Choose measures before you start

Select two or three measures that are practical to collect. Examples include the change in rubric scores from a first draft to a revision, scores on a comparable end-of-unit writing task, and the percentage of students who act on at least one comment. Decide how they will be scored and by whom. Setting these in advance prevents the temptation to cherry-pick results.

• Pick two or three measures such as rubric change between drafts and later task scores
• Decide how the measures will be scored before the term begins
• Compare sections or use a switch design so every class benefits
• Rate a sample of comments on accuracy, specificity, and actionability
• Ask students what they did with the feedback they received

Create a fair comparison

If possible, compare similar groups of students. One option is to use the new approach in some sections while others continue as usual, then switch halfway through the term so that every class benefits. Another is to compare this year's results with last year's for the same assignment and course. Neither is perfect, but both give you more than impressions.

Pay attention to differences that could distort the comparison, such as differences in class composition or in how teachers implement the approach. Keep notes on what actually happened in each section. Share them when you present results. Honest limits make findings more believable, and a short written note on class sizes, schedules, and implementation helps readers interpret the numbers correctly and prevents overconfident conclusions.

Look at the quality of the feedback itself

Sample a set of comments from each group and rate them on accuracy, specificity, and actionability. Have two teachers rate independently and compare. High-quality feedback is a necessary condition for improvement, and weak comments will undermine any approach. This analysis can reveal whether the feedback process, rather than the tool alone, is responsible for results, and sharing a few anonymized examples of strong and weak comments with the whole department makes the standards concrete for everyone.

Ask students directly what they found helpful and what they did with the comments. Short surveys and focus groups often reveal that students use feedback differently from what teachers expect. Pay attention to whether students understood the comments and whether they felt respected by them. Student voices round out the numbers, and even a five-question form, completed in class, can supply information that no amount of score analysis could uncover on its own.

Make decisions and share findings

After the measurement period, convene a small group to review the data and discuss implications. Consider both the benefits and any risks, including workload for review, equity concerns, and privacy. Decide whether to continue, adjust, expand, or stop. Write down the reasoning, and including a teacher who was skeptical of the approach in that group tends to produce a more balanced reading of the evidence, which strengthens the final decision for everyone involved.

Share a brief summary with staff, families, and the board where appropriate. Include what you measured, what you found, and what you will do next. Transparency strengthens trust and models the careful use of evidence you hope students will learn. Repeat the process when circumstances change, and a one-page summary with two charts and a short list of next steps is usually enough to keep the decision transparent and the process credible.