How Do AI Detectors Work? Accuracy, Limits and False Flags

Disclosure: Some links in this post are affiliate links. If you buy through them, we may earn a commission at no extra cost to you. Learn more.

AI detectors work by estimating how "predictable" a piece of writing is. They run your text through a language model (or a classifier trained on human and AI samples) and score how closely your word choices and sentence patterns match what an AI would typically produce. The result is a probability, not proof, and that's why detectors get it wrong more often than most people think.

This guide explains how AI detectors work step by step, what perplexity and burstiness mean, how accurate the big tools say they are, and what to do if your own writing gets flagged.

How do AI detectors work? The short version

Every AI text detector, whatever the brand, does some version of three things:

Breaks your text into pieces (sentences or chunks of words, called tokens).
Scores each piece for how likely it is that a language model would have written exactly those words in that order.
Combines the scores into an overall estimate, usually shown as a percentage or a label like "likely AI-written."

What changes between tools is how they score. Older and cheaper detectors lean on simple statistics. Newer ones use deep-learning classifiers trained on huge sets of human and AI writing, and some combine several signals.

Perplexity and burstiness, explained simply

These two terms come up in almost every explanation of AI detection. GPTZero, one of the best-known detectors, built its original model on them.

Perplexity: how surprising are your words?

Language models write by predicting the next most likely word. GPTZero describes perplexity as a measure of "how likely an AI model would have chosen the exact same set of words" found in your document.

GPTZero's own example: the sentence "Hi there, I am an AI ___" would most likely be finished with "assistant." That's low perplexity, very predictable. If the next word were "potato," perplexity would be high, and the sentence would look more human. Over hundreds of words, these small probabilities add up into an overall picture.

Burstiness: does your rhythm vary?

Humans write unevenly. We mix a long, winding sentence with a short one. Then a fragment. Burstiness measures how much that predictability changes from sentence to sentence. AI text has tended to be more uniform, while human text tends to spike and dip.

Why these signals alone aren't enough

GPTZero notes that these statistical methods are cheap and fast, and that they sit behind many other detector apps. But it also says that as of autumn 2023 it no longer relies on perplexity and burstiness as its main method. It moved to a deep-learning architecture and treats the statistical signals as one input among several. The lesson: simple predictability scores are a weak test on their own.

The 3 main types of AI detection

Method	How it works	Strengths	Weaknesses
Statistical scoring	Measures perplexity, burstiness and similar patterns with a language model	Fast, cheap, easy to explain	Flags simple, formal or formulaic human writing
Trained classifiers	A model trained on labeled human and AI text predicts which one a new text looks like	Catches subtler patterns	A black box; struggles with new AI models and edited text
Watermarking / provenance	The AI tool hides a statistical signal in its output, or records where content came from, so it can be checked later	Can be far more reliable when it's present	Only works if the AI tool added it, and editing can weaken it

Most commercial detectors today are trained classifiers, sometimes mixed with statistical signals. Watermarking is a different approach: instead of guessing after the fact, it builds a detectable signal into the text at the moment it's generated. OpenAI, for example, said when it retired its own detector that it was "researching more effective provenance techniques for text."

How accurate are AI detectors?

This is where it gets uncomfortable. Here's what the companies and institutions themselves have published.

OpenAI shut down its own detector

In January 2023, OpenAI released an AI text classifier. Its own launch post said it was "not fully reliable": on OpenAI's challenge set it correctly flagged only 26% of AI-written text as "likely AI-written" and wrongly labeled human writing as AI 9% of the time. On July 20, 2023, OpenAI withdrew it "due to its low rate of accuracy."

Turnitin's published false positive rates

Turnitin, widely used by schools, says its document-level false positive rate (fully human text wrongly marked as AI) is less than 1% for documents with 20% or more AI writing. Its sentence-level false positive rate is around 4%, meaning there's about a 4% chance a specific highlighted sentence was actually written by a human. Turnitin adds that these errors are more common in documents that mix human and AI writing, and it tells instructors to use the score "to initiate a conversation, not to draw a conclusion."

Why "less than 1%" still matters

Vanderbilt University did the math when it decided in August 2023 to disable Turnitin's AI detector. Vanderbilt submitted 75,000 papers to Turnitin in 2022. At a 1% false positive rate, it estimated, around 750 student papers could have been wrongly labeled as partly AI-written. Vanderbilt also cited concerns about how little Turnitin explained about how the tool works.

Bias against non-native English writers

A Stanford-led study, "GPT detectors are biased against non-native English writers" (Liang et al., 2023), tested several popular detectors. The researchers found the detectors "consistently misclassify non-native English writing samples as AI-generated," while native writing samples were identified accurately. They also showed that simple prompting could get around the detectors, and cautioned against using them in evaluative or educational settings.

The reason ties back to perplexity: writers with a smaller vocabulary, or who stick to safe, standard phrasing, produce text that looks "predictable," which is exactly what detectors are trained to flag.

What makes AI detectors give false positives?

Based on how detectors work, these kinds of human writing are most at risk:

Formal or formulaic writing: lab reports, legal text, product descriptions and five-paragraph essays follow predictable patterns.
Writing by non-native English speakers, as the Stanford study found.
Short texts: OpenAI noted its classifier became more reliable as input length increased. Fewer words means less evidence.
Heavily edited text: grammar tools that smooth every sentence can make writing more uniform.
Mixed documents: Turnitin says sentence-level errors are more common where human and AI writing meet.

What to do if your writing is flagged as AI

If you wrote it yourself and a detector says otherwise, stay calm and gather evidence:

Show your process. Version history in Google Docs or Word, outlines, notes and earlier drafts are the strongest evidence you have.
Ask what the score actually means. Turnitin itself says there's no "right" or "target" score and that a highlight is a reason to talk, not a verdict.
Offer to explain the work. Turnitin recommends instructors ask students to walk through their ideas and process. If you wrote it, this is easy.
Point to the published limits. OpenAI's withdrawal, Turnitin's own sentence-level error rate and the Stanford study are all public. Links are in the Sources section below.

If you're a teacher, editor or client, use detectors as one signal at most, never as the only reason for a penalty.

Should you use an AI detector at all?

They have a place, as long as you treat the result as a hint:

Content managers can use a detector to decide which freelance drafts deserve a closer read.
Writers can run their own work through one to spot passages that read flat and generic, then rewrite those for clarity and voice. That's a good editing habit anyway.
Educators get more from clear AI-use policies and process-based assignments (drafts, reflections, oral check-ins) than from a percentage.

Some writing tools bundle detection now. Grammarly lists "Detect plagiarism and AI generated text" as a feature of its paid Pro plan, and QuillBot's free plan includes limited access to its AI detector. We compare those writing tools in our guide to Grammarly vs free alternatives.

FAQ

How do AI detectors work in simple terms?

They use a language model or a trained classifier to estimate how predictable your text is. If your word choices and sentence patterns look like what an AI would typically write, the detector raises its "AI" score. The output is a probability, not proof.

Are AI detectors accurate?

Not reliably enough to prove anything on their own. OpenAI withdrew its detector in 2023 for low accuracy, Turnitin reports a sentence-level false positive rate of around 4%, and a Stanford-led study found detectors often misclassify non-native English writing as AI.

What is perplexity in AI detection?

Perplexity measures how surprising your word choices are to a language model. Low perplexity means the text is very predictable, which detectors associate with AI. High perplexity means more unexpected word choices, which they associate with humans.

Can AI detectors detect ChatGPT?

They often flag unedited ChatGPT text, but results vary by tool, text length and how much the text was edited. The same study that found bias against non-native writers also showed simple prompting could bypass detectors.

Why was my own writing flagged as AI?

Common reasons are formal or formulaic style, short length, heavy grammar-tool editing, or writing in English as a second language. Keep your drafts and version history so you can show how you wrote it.

Conclusion

AI detectors don't "know" who wrote something. They measure how predictable text looks and turn that into a probability. That's useful as a hint and dangerous as a verdict, as OpenAI's withdrawn classifier, Turnitin's own error rates and the Stanford bias study all show. Write in your own voice, keep your drafts, and treat any score as the start of a conversation.

For more clear, no-hype explainers on the AI tools everyone's talking about, subscribe to AI Money Pilot on YouTube.