When a vendor says a tool has a one percent false-positive rate, the number sounds reassuring. Vanderbilt University looked at what that figure would mean in practice and calculated that, across roughly 75,000 student submissions in a year, about 750 pieces of genuine work could be wrongly labeled as AI. That is 750 students facing questions, stress, and possible penalties for something they did not do. The same arithmetic applies to any screening process, whether it detects plagiarism, flags struggling writers, or draws attention to unusual scores.

The mistake is to treat a percentage as if it applied to one student at a time. A one percent error feels like a one in a hundred chance, but the real question for an institution is how many people will be affected in total. Large numbers turn small rates into real harm. This is why careful organizations consider both the rate and the volume.

Base rates add another layer. If only a small share of submissions actually involve misconduct, then even an accurate tool will produce many false alarms compared with true catches. In some settings, a flagged paper may be more likely to be innocent than guilty. Understanding this helps educators resist the temptation to treat any flag as strong evidence, and that is why careful institutions require a human investigation before any consequence follows from a flag.

Work through an example with your own numbers

Imagine a high school with 1,200 students, each submitting five essays a year, for 6,000 submissions. A tool with a one percent false-positive rate would wrongly flag about 60 genuine essays. If it catches most real cases but only a few dozen essays are actually AI-generated, a large share of flags would be wrong. Presenting this calculation to staff helps them see why a flag cannot be the end of the analysis.

• Multiply any error rate by the number of submissions in a year
• Ask what share of flagged work is likely to be genuinely problematic
• Request error rates broken down by student group and writing type
• Require corroborating evidence and a student response before any action
• Track how many flags or score changes turn out to be mistakes

Remember that errors are not evenly spread

The overall rate can hide large differences between groups. Independent studies have found much higher false-positive rates for essays by multilingual writers than for native English speakers. A tool that is quite accurate overall may be unreliable for the students least able to bear an accusation. Ask for results broken down by group, and test the tool on your own population if you can.

The same principle applies to automated scoring. A grading tool that agrees with teachers most of the time may still have patterns of disagreement concentrated in specific kinds of writing or student groups. Overall agreement statistics should be accompanied by a look at where the errors fall. Fairness depends on the details, not just the average, and asking a vendor for agreement results by student group is a reasonable and increasingly common request.

Build procedures that expect errors

If you use any screening tool, design the process on the assumption that some flags will be wrong. Require corroborating evidence before any action, give students a meaningful chance to respond, and keep records of outcomes. Track how many flags are confirmed and how many are cleared, which tells you how useful the tool really is. If most flags turn out to be unfounded, the tool is creating more work than it saves.

For grading support tools, the equivalent practice is teacher review. Because every draft score passes through a person who can correct it, errors are caught before they affect students. Track how often teachers change scores and where. That information is the practical error rate in your setting, and a quarterly summary of those changes, shared with the department, is a simple way to monitor quality over time.

Communicate the numbers honestly

When you talk to students, families, and colleagues about screening or scoring tools, be willing to share the limits. A simple statement that no tool is perfect, that errors are expected, and that people review every decision builds more trust than a promise of accuracy. Include the human safeguards in your explanation. Honesty about uncertainty is a strength, and families in particular tend to respond well when they learn exactly where a person steps in to check the work.

Encourage vendors and administrators to publish error rates and testing methods in plain language. Ask what populations and writing types were included in testing. Decisions made with a clear understanding of the numbers are better decisions. The arithmetic is simple, but it is often overlooked, and a short list of questions, asked consistently of every vendor, makes comparisons easier and decisions more defensible.