Why do AI answers change every time you ask the same question?

The experiment

On 29 August 2026 we ran two public AI visibility indexes — appointment scheduling tools and podcast hosting platforms — and then, a few hours later, ran both again. Same questions, same wording, same five AI assistants: ChatGPT, Claude, Grok, Perplexity and Google AI Overviews. Eight buyer questions per category, giving about forty usable answers per run.

Nothing about any of the brands changed in those few hours. No pricing pages moved, no launches happened, no content shipped. Whatever difference appeared between the two runs was the measurement, not the market.

The difference was large.

What moved

We compared twenty brands across the two categories. Fifteen of them were named a different number of times in the afternoon than in the morning, and ten changed ranking. The largest movements:

None of it was real

A change in a count is only evidence if it is bigger than the count wobbles on its own. The standard test for two proportions gives each movement a z-score, and the conventional bar for calling something real is 1.96 — roughly, a result that chance would produce less than one time in twenty.

The largest movement in the set, Setmore falling three places, scores 1.25. Acast reaches 1.01. Everything else is below 0.8. Not one of the twenty changes clears the bar, and most are not close.

The consequence is worth stating plainly, because it is the whole point. Buzzsprout overtook Spotify for Creators for the top spot in podcast hosting. In scheduling, Cal.com went from fifth to second and Setmore fell from second to fifth, passing each other. Nothing happened. A screenshot of either run would have supported a confident story about momentum in that category, and the story would have been about nothing at all.

Why it happens, and why bigger samples are the only cure

Two sources of variation stack. The model itself is probabilistic: at each step it samples from a distribution over possible next words, so two runs of the same prompt genuinely diverge rather than repeating. On top of that, AI assistants that search the web fetch a live result set that changes between runs — different sources in, different brands out.

The reason this shows up so violently in the table above is sample size. Eight questions across five AI assistants is forty draws. For a brand named in about half of them, ordinary sampling variation is worth roughly nine answers in either direction before anything unusual has happened. Six answers of movement, which looks dramatic on a chart, is inside that.

Repetition helps less than people expect. Asking the same question ten times measures how unstable that one question is; it does not tell you about the nine questions you did not ask. Reliability is bought by breadth across questions and AI assistants far more cheaply than by depth of repetition on a few.

The honest counter-evidence

It would be convenient to conclude that single-run sampling is worthless. The evidence says otherwise, and it comes from a competitor.

Profound published an A/B test comparing one run per prompt per day against ten, across roughly 5.78 million runs. Their finding: once a day lands within about two percentage points of a 10× reading for visibility, and citation share differed by around 0.3 points. If you are averaging across thousands of prompts, run-to-run noise largely cancels, and a daily sample is a reasonable estimate of the aggregate. That is a real result and anyone comparing tools should read it.

It does not contradict what we found; the two are answers to different questions. Variance cancels across a portfolio of thousands of prompts. It does not cancel within a single row, or across the forty answers most small businesses are actually buying. “Are we in the answer for best scheduling tool for a small business?” is one prompt. Asked once, the result is a sample from a distribution, and you cannot tell a stable absence from an unlucky one.

What holds still when the numbers don't

The useful finding is not that measurement is hopeless. It is that the score was never the durable part. Underneath the number, three things about a brand are stable across runs even while the count jumps around:

Questions you are named in every single time. Questions you are never named in. And questions where the AI assistant names you sometimes — where it plainly knows who you are and is choosing between you and someone else, run to run.

That third group is the only place a content change has observable leverage, and it is invisible in a single score. A brand can hold a steady score for a month while the sometimes-named set quietly shifts underneath it. The states are also far more robust than the count: a question you win 10 times out of 10 does not become a question you win 4 times out of 10 because of sampling.

• Always named — you were in every answer. Nothing to fix; this is what you protect.
• Sometimes named — you were in some answers and not others. Where a change is most likely to pay, because you are already in the running.
• Never named — you were in none of the answers. Needs a new page or source, not a wording change.

How to compare two runs properly

Comparing two aggregate percentages throws away the most useful fact about the data: both runs asked the same questions of the same AI assistants. Most of those question-and-assistant pairs never move — a category leader is named in essentially everything, every time. The variation lives in a minority of cells, and once the stable ones are held fixed, a much smaller change becomes detectable.

Testing only the cells that actually flipped, and asking whether they lean one way further than a coin would, roughly halves the change a given sample size can resolve:

What we changed in our own product

This experiment was not a study we set out to run. We ran an index twice because we had fixed something else, and the second run disagreed with the first badly enough to be alarming. Two of our own features were wrong, and both had shipped.

Our public index pages drew a movement chip beside every brand — a red down-arrow beside Setmore, a green up-arrow beside Cal.com — computed by subtracting one run from the other with no test at all. Every one of those chips on 29 August was false. They now render as unchanged unless the movement beats the noise, with the raw figure available on hover.

Worse, our tracking alerts fired on any change whatsoever: the rule was literally “if the count is different”, applied separately to overall presence, to citations, and to each of five AI assistants. That afternoon’s noise would have generated a headline, a presence alert, a citation alert and up to five per-assistant alerts, every one of them false, on a weekly schedule, forever. Every reported change is now tested before it is sent.

We are describing our own bugs in public because the alternative is to publish a study about other people’s numbers being unreliable while quietly having shipped the same error. And because the fix is the interesting part: a tool that measures its own variance can tell you which changes are real, and one that does not is guessing on your behalf.

What to ask any tool that sells you a number

There is one question that separates a measurement from a decoration, and almost nothing on the market answers it.

• How many times did you ask? A number without a sample size is not checkable.
• What is the smallest change you can detect? If a tool cannot answer this, it cannot know whether the movement it is reporting is real.
• Do you test a change before alerting me? Subtracting two numbers is not a test.
• Can I see the answers behind the count? A count is only auditable if the text it came from is kept.

Common questions