Auditing What AI Gets Factually Wrong About a Brand

Ninety-one percent accurate is a number you cannot act on

In late 2025 and early 2026 the AI startup Oumi ran a fixed benchmark against Google's AI Overviews for the New York Times. Oumi reran the same experiment in both periods, once on Gemini 2 and again after the switch to Gemini 3. Search Engine Land reported the scale and the result: 4,326 searches, with accuracy rising from 85 percent to 91 percent.

Then Oumi checked whether the cited pages actually supported the answers. Only 39% of the total overviews were both correct and fully supported by their citations. Only 67 percent of individual claims were supported at all.

Two different failures hide inside the word accuracy. The engine can state something false. The engine can also state something true that nothing retrievable supports. Those have opposite fixes, and a single headline percentage tells you which one you have exactly never.

That is the flaw I want to fix here, and it is not a Google flaw. It is an audit design flaw, and it is ours.

Three reasons your last accuracy audit cannot be re-run

I have looked at a lot of these. They fail in the same three places.

No fixed field set. The auditor scores whatever the engine happened to say. One run catches a wrong phone number, the next catches a wrong founding year, and the two runs are not comparable. You cannot show improvement between two audits that measured different things.

No named adjudicator. The audit calls a claim wrong because the client said so. That is a complaint, not a finding. If the party being measured is also the referee, the score is decoration.

One sample per prompt. This is the one that quietly destroys everything. A 2026 variance-components study of 12,933 LLM brand responses across three models and eight languages found that within-prompt resampling alone accounts for 34.8% of the variance in brand answers. Ask the same question twice and you get materially different output, at temperature 0.3, with nothing else changed.

The same paper gives the practical stopping rule. A repeat past the fifth reduces relative-error variance by roughly 0.0003. Five runs per prompt is not a hedge, it is the point where more runs stop buying you anything. That is a better answer than any vendor has published on how many samples an AI visibility reading actually needs.

One more constraint, from a companion study of 3,750 responses across 50 brands and five industries: cross-model agreement on the top-recommended brand was 41.6%. Never pool engines into one score. Score ChatGPT, Gemini and Perplexity separately or you are averaging three different systems into a number that describes none of them.

The eight fields, and the public source that settles each

A field earns its place only if somebody other than the brand can settle it. That single rule cuts the list down fast, and what survives is boring, checkable, and durable.

Rank them by consequence, not by how often they break. Licensing status and phone are the two that cost real money the day they are wrong, because one invites a regulatory complaint and the other routes a buyer to a dead line. Founding year is the one clients care about most and the one that matters least.

Field 2 needs an honest caveat, because I have seen it fudged. A registration date is a proxy, not a birth certificate. Businesses reorganise, buy predecessors, and operate for years as sole proprietors before incorporating. If the brand claims an earlier founding, the audit accepts it only when the evidence sits on a page a crawler can reach. If the evidence lives in a filing cabinet, the engine was never going to find it, and the field fails.

Field 7 has a jurisdictional hole worth naming. Plenty of states do not publish officers for privately held entities. Where the registry is silent, the correct score is not "pass," it is "unadjudicable in this jurisdiction," and the field drops out of the denominator for that brand. Shrinking the denominator honestly beats inflating the score.

The two fields I cut, and why the exclusions carry the credibility

I wanted ten. I could only defend eight.

Price band, excluded. There is no public authority on what a service costs. The only source is the brand's own pricing page, and the brand is the party under audit. Score it and you have built a mechanism where the audited party grades its own paper.

Stated differentiator, excluded. "Fastest response in the county." "Most experienced team in Southern Utah." These are positioning claims. They are not false, they are unfalsifiable, which is worse for a metric. An engine that repeats your differentiator has not been accurate, it has been persuaded, and that belongs in a recommendation reading rather than a citation or accuracy reading.

Keep both rows visible on the scoring sheet, greyed out and stamped excluded. The visible exclusion is the part that makes the other eight believable. A scoring instrument with no declared out-of-scope is almost always a sales instrument.

This is the same discipline that separates a defensible client report from a vanity dashboard, which is why the exclusions belong in what you actually send the client, not just in your working file.

The run protocol

One hundred and twenty observations sounds heavy until you run it. It is roughly ninety minutes of work for a single-location business, and it is the only version of this that produces a number you can put next to the same number in October.

Scoring in three states, not two

The distinction between wrong and silent is where most audits leak credibility, and the research says which way the leak runs.

The Tow Center at Columbia tested eight generative search tools across 1,600 queries and found they returned incorrect answers to more than 60 percent of them, with Perplexity at 37 percent wrong and Grok 3 at 94 percent. The behavioural finding matters more than the headline rate.

Read that as an audit design instruction. Engines are structurally biased toward answering, so "not stated" will be rarer than your intuition says and "incorrect" will be more common. If your audit is returning a pile of blanks, suspect your prompts before you celebrate.

The grounding column has its own literature and it is older than most people assume. Nelson Liu, Tianyi Zhang and Percy Liang measured four generative search engines for EMNLP 2023 and found a mere 51.5% of generated sentences are fully supported by citations, with 74.5 percent of citations supporting the sentence they were attached to. Three years on, Oumi's 67 percent claim-support figure sits in the same neighbourhood. The grounding problem has not been engineered away, so measure it as a standing column rather than as a one-off curiosity.

Use this graphic on your site

Free to republish with a link back to this page. Copy the embed code:

<a href="https://josephtimpson.com/insights/brand-accuracy-in-ai-answers"><img src="https://josephtimpson.com/assets/infographics/brand-accuracy-in-ai-answers.svg" alt="Checklist of eight objectively checkable brand fields with their public sources of truth, plus two excluded fields" width="1200" style="max-width:100%;height:auto"></a><p>Graphic by <a href="https://josephtimpson.com/insights/brand-accuracy-in-ai-answers">Joseph Timpson</a></p>

There is also no first party number to check yourself against. No engine publishes your accuracy rate. Everything here is sampled, and sampled measurement carries error bars whether or not the tool draws them, which is the standing caveat on every AI visibility tracking tool on the market.

He wrote that about local search, and it travels. Nobody auditing AI answers has privileged access either. The eight-field score is defensible because its inputs are public and its procedure is repeatable, not because it sees inside anything.

Fixing what the audit finds, in order

The repair order is fixed, because the cheap fixes are also the ones that unblock the expensive ones.

Start with retrievability. Google states that to appear in its generative features, a page must be indexed and eligible to be shown with a snippet. A stray nosnippet tag can zero out every field at once, which is why a crawler access check runs before an accuracy audit, not after it.

Second, put the eight facts in plain text on a page. Not in an image, not in a PDF, not only in markup. Google says outright that structured data is not required for generative AI search, and Ahrefs' controlled test on 1,885 pages found no meaningful citation uplift from adding schema. Mark it up anyway for classic search hygiene, but do not sell it as the accuracy fix. It is not, and the evidence against schema as an AI lever is now hard to argue with.

Third, correct the upstream records: the state registry, the licence board, the Google Business Profile, and the directories that syndicate them. This is unglamorous entity maintenance and it is where the incorrect state usually gets its material, which is the practical half of any real entity SEO programme.

Fourth, and slowest, work the third-party corpus. Across 15 SaaS brands, 84% to 93% of AI citation weight sat on third-party sites rather than the brand's own domain. If a review site, an old press release or a competitor's comparison page carries a wrong fact about you, your own site correcting it is necessary and often not sufficient. That is the same mechanism behind why brand mentions out-correlate backlinks, running in reverse.

Audiences are already doing this cross-checking themselves. BrightLocal's 2026 research found 45% of consumers have used AI tools to find local business recommendations, up from 6 percent the year before, and that 97 percent of AI users at least sometimes cross-check those recommendations against real reviews. A wrong fact in an AI answer does not just misinform, it fails the verification step the buyer runs next.

Re-running it, which is the entire point

Same eight fields. Same prompts, word for word. Same three engines. Same five repeats. Sixty to ninety days later.

Report four numbers per engine: correct, incorrect, not stated, and grounded. Movement in the incorrect column is the only one that proves a fix landed, because a field moving from incorrect to not stated means the bad source stopped winning but nothing replaced it. That is progress, and it is not victory.

Do not celebrate a swing smaller than the noise floor. With 34.8 percent of variance coming from resampling alone, a two-field improvement on a forty-observation-per-engine sample is inside the error bar. Say so in the report. Clients trust the practitioner who names the noise, and they eventually stop trusting the one who does not.

If you want the fuller apparatus this sits inside, the method is written up here, and the rest of the instrument set lives in the insights index. The next artifact to build against this one is a worked audit on a real brand, where these eight fields get run end to end and the failures get published alongside the passes.