AI Visibility Scores Are Mostly Noise. What to Measure Instead

Every AI tracking tool on the market right now wants to sell you the same thing. A number. Your brand scores a 73 out of 100 for AI visibility, your competitor scores a 61, and for a monthly fee you can watch those numbers wiggle around on a dashboard.

That number is mostly noise. Not because AI visibility doesn’t matter. It matters more every quarter. The number is noise because of how it gets measured, and once you understand the measurement problem, you’ll never look at a single-shot visibility score the same way again.

Ask the Same Question Five Times. Count the Different Answers.

Old Google worked like an index. You searched a keyword, Google returned a ranked list, and position 3 stayed position 3 until something changed. You could measure it because there was a fixed thing to measure.

AI answers don’t work that way. Ask ChatGPT for the best CRM for a small law firm, then ask again ten minutes later. Different response. Sometimes the same brands in a different order, sometimes a brand appears in run one and vanishes in run two. I’ve watched client mentions flicker in and out across repeated runs of an identical prompt. Nothing changed on the web in those ten minutes. The model just generated a different answer, because generating different answers is what these systems do.

So when a tool runs your target prompt once, records whether you showed up, and converts that into a score, it’s reporting a coin flip as data. You didn’t score a 73. You got heads on the particular flips they happened to run that day.

The Prompt Represents a User Who Doesn’t Exist

There’s a second problem stacked on top of the first one. The prompts these tools track are generic on purpose. “Best CRM 2026.” “Top roofing company near me.” Clean, trackable, and representative of absolutely nobody.

Real buyers don’t prompt like that. They ask with constraints, with history, with context the model can see and your tracking tool can’t. I wrote about this a couple weeks back when Google’s Personal Intelligence testing showed Gmail contents shifting which brands AI Mode recommends. If the model is reading a user’s inbox before answering, then the “average user” your tracking tool simulates literally cannot exist. Every real user gets a personalized distribution. Your score measures a ghost.

More Prompts Doesn’t Fix It. It Just Costs More.

The industry’s answer to this critique is already visible: add more prompts. Track five phrasings across three intent levels across four buyer personas and suddenly you’re running 60 prompt combinations per keyword before anyone adds geo modifiers or industry variants.

Run the math on what that costs at scale and you’ll understand why the dashboards keep getting more expensive. But volume doesn’t repair a representativeness flaw. Sixty flawed measurements aren’t better than one flawed measurement. They’re the same flaw with a bigger invoice attached.

The Part Nobody Selling Dashboards Wants to Say Out Loud

Now the other side of it, because “AI answers are random, nothing is measurable, panic” is also wrong.

AI answers are not pure vibes. Perplexity, AI Overviews, and ChatGPT with search all run a retrieval layer. Before the model writes a word, it pulls live pages off the open web and grounds its answer in them. That retrieval step behaves a lot more like traditional search than the probabilistic output does. Pages get fetched because they rank, because they carry authority, because they’re structured in a way the engine can extract.

That layer is the part you can influence. It’s why indexation and link equity still decide who shows up in AI answers, the same way they decide who shows up on page one. The generation step is probabilistic. The retrieval step rewards structural authority. Brands that dominate retrieval show up in more runs. Brands that don’t, flicker.

Which points directly at the metric that actually holds up.

Appearance Rate Is the Honest Metric

If AI output is a distribution, measure it like one. Take the questions your actual buyers ask, staged by intent from research to comparison to purchase. Run each question multiple times against each engine. Report how often you appear.

“Cited in 2 of 3 runs” is a real finding. It tells you your retrieval presence is strong enough to surface most of the time but not locked in. “Cited in 0 of 3 runs” across an entire query set is a real finding too, and a brutal one. I’ve delivered that report. There’s no arguing with a zero that repeats.

A 73 out of 100 tells you none of that. It’s decoration on top of a coin flip.

This is exactly how I run AI citation work on client accounts now. Every query gets multiple runs per engine, and the report shows appearance rate per question, not a blended score. The first time I switched a client from a single-run snapshot to repeated runs, two “visible” queries turned out to be one-in-three flukes. The score said fine. The distribution said fix your retrieval footprint.

The dashboards will keep selling numbers because numbers renew subscriptions. Measure the distribution instead. Then go do the unglamorous work that moves it, which was never a mystery: rank the pages the engines retrieve, hold the authority that gets you retrieved, and stop paying for decimal points of noise.