I could not reproduce the 40% GEO result. Here is exactly why.
I published my acceptance criteria before I opened anything, then audited the founding GEO paper against its own released code and dataset. One of five criteria cleared. The audit also found something in the source code that changes how you should read the headline.
- The famous 40% is a 30 to 40 percent relative gain on Position-Adjusted Word Count, an internal metric that no production engine exposes and that is not a citation count.
- The paper's own text reports a 115.1% visibility gain for the fifth ranked source alongside a 30.3% average loss for the top ranked source, meaning it measured redistribution inside a fixed five source answer, not new visibility.
- Reproduction fails on four of five criteria: the engine was gpt3.5-turbo, the sources were frozen 2023 Google top-5 results, the metric is unmeasurable in a live engine, and the judge was GPT-3.5 with no human annotators for the main results.
- The three winning tactics were implemented by prompting GPT-3.5 to add hypothetical statistics, admittedly fake quotes and invented sources. The released code says so in plain English.
- What survives best is the paper's negative result: keyword stuffing scored 17.7 against a 19.3 unoptimized baseline, worse than doing nothing.
The number everyone quotes is an average of a zero sum game
The GEO paper is real, peer reviewed at KDD 2024, and honestly reported by its authors. It measured a 30 to 40 percent relative gain on one internal metric, inside a frozen five source answer, scored by GPT-3.5. It never measured citations and it never measured traffic. Of the five conditions a meaningful 2026 retest needs, one cleared.
The paper is GEO: Generative Engine Optimization, by Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan and Ameet Deshpande. Posted November 2023, revised June 2024, accepted to KDD 2024.
It is good work. It is also the only academic anchor most GEO pitch decks have, which is why what it actually says matters more than usual.
The abstract compresses the result into visibility boosted "by up to 40%". The body is narrower and better. In the authors' own words, the top three methods "achieved a relative improvement of 30-40% on the Position-Adjusted Word Count metric."
Then there is the sentence nobody quotes. From the full text of the paper: "the Cite Sources method led to a substantial 115.1% increase in visibility for websites ranked fifth in SERP, while on average, the visibility of the top-ranked website decreased by 30.3%."
Read that twice. Every answer in the benchmark drew on exactly five sources, and the paper normalizes its impression metrics "by multiplying them with a constant factor so that the sum of the impressions of all citations in a response equals 1." Optimization moved share away from the incumbent source and toward the challenger source.
That is redistribution inside a closed set. It is not the creation of new visibility, which is what "40 percent more AI visibility" means to the person signing the invoice. If you want the taxonomy of how these claims drift, I keep the running list in the GEO myths audit.
Our top-performing methods, namely Cite Sources, Quotation Addition, and Statistics Addition, achieved a relative improvement of 30-40% on the Position-Adjusted Word Count metric.
The five things that had to be true, written down before I opened anything
A replication that sets its own pass mark after seeing the result is not a replication. So here are the criteria, in the order I wrote them, before I touched the repository.
For a 2026 retest to be a comparison rather than a new study wearing a borrowed headline, all five had to hold.
- The benchmark queries and their attached source documents must be obtainable unchanged.
- The generative engine must be the same model, or a defensible stand-in with a stated mapping.
- The retrieval step must work the way the original worked, or the difference must be quantifiable.
- The outcome metric must be computable on a production engine, not only inside a simulator.
- The evaluation judge must be the same, or its replacement must be calibrated against the original.
I added one disqualifier. If the only way to get a result was to change the metric, the retest would be reported as a new experiment and not as a replication, because swapping the dependent variable is how honest people accidentally publish nonsense.
One criterion cleared. Here is the scorecard, then the detail.
| Criterion | Result | Blocker |
|---|---|---|
| Queries and sources obtainable | Pass | None. Public and versioned. |
| Same generative engine | Fail | gpt3.5-turbo, a retiring model family |
| Same retrieval | Fail | Frozen 2023 Google top-5, no fan-out |
| Metric computable live | Fail | Position-Adjusted Word Count is simulator only |
| Same judge | Fail | GPT-3.5 running G-Eval, no human annotators |
What reproduced: the artifacts, and credit where it is due
This is the part that went better than expected.
GEO-bench is publicly hosted on Hugging Face under CC BY-NC-SA 4.0, split 8,000 train, 1,000 validation, 1,000 test, for 10,000 queries total. Each row carries the query, its tags, and the sources. The card states that alongside the queries the authors "also provide 5 cleaned html responses based on top Google search results."
The harness is public too. The GEO repository is Apache-2.0, with 312 stars and 62 forks at the time I looked, and src/run_geo.py loads the frozen test split rather than hitting a live search API. Last commit October 30, 2025.
So you can rerun the lab. That is real openness and it is rarer than it should be.
Here is the ranking that matters: on artifact hygiene, this 2023 academic paper beats almost every vendor study the same industry quotes without blinking. Most vendor studies in this space publish a methodology summary and keep the underlying data private. Aggarwal and his co-authors published the queries, the sources, the prompts and the runner. When I write about how to judge an AI visibility study, this is the standard I hold vendors to.
And it still does not get you a replication, because the lab is not the world.
What did not reproduce, in four parts
The engine is a retiring model
The paper states it plainly: "The answer is then generated by the gpt3.5-turbo model using the same prompt as prior work." Temperature 0.7, five sampled answers per condition.
OpenAI has been walking that family out the door. Its deprecations page lists a 2025-09-26 announcement retiring gpt-3.5-turbo-instruct on 2026-09-28, with gpt-5.4-mini or gpt-5-mini as the recommended replacement, on a page dense with 2026 shutdown dates.
You can substitute a current model. You cannot then claim you reproduced anything, because the thing being measured is a model's preference, and you swapped the model. The dependent variable and the apparatus are the same object here.
The retrieval step is frozen 2023
From the paper: "only the top 5 sources are fetched from the Google search engine for every query." Fetched once. Cleaned. Stored.
Production engines stopped working that way. Google's own documentation on optimizing for generative AI features describes grounding plus a query fan-out technique, meaning one user question becomes many retrieval queries. Ahrefs, analysing 1.4 million ChatGPT prompts, found cited URL titles matched ChatGPT's internally generated fan-out queries more closely (cosine 0.656) than they matched the user's actual prompt (0.602).
The benchmark's competitive set is five fixed documents. A live engine's competitive set is whatever a dozen subqueries drag in. Those are not the same experiment, and I unpack the mechanism separately in the query fan-out breakdown.
The metric is not a citation
Position-Adjusted Word Count measures how much of the generated answer's text came from your source, weighted by where it appeared. It needs the full source set, including the sources that lost, and per source word attribution inside the answer.
No production engine exposes any of that. You see which URLs got linked. You do not see which twenty did not.
The strongest confirmation of this problem is academic. A March 2026 arXiv paper, Diagnosing and Repairing Citation Failures in Generative Engine Optimization by Zhihua Tian and colleagues, opens by stating that existing methods measure contribution rather than "citation, the mechanism that actually drives traffic back to creators." That is the field itself saying the original metric was not the thing marketers think they bought.
This is also why I keep insisting that citations and recommendations are different measurements. Conflating them is the same error at a smaller scale.
The judge is the engine
The subjective half of the evaluation used G-Eval with GPT-3.5. The paper is explicit that the sub-metrics are "evaluated using GPT-3.5, using methodology similar to described as in G-Eval." Human annotators appear only for verifying benchmark tags, not for scoring the main results.
So the model that wrote the answers is closely related to the model that graded them. That is a defensible research choice in 2023 and a fatal one for a 2026 comparison, because your substitute judge has different taste and you cannot separate a change in visibility from a change in the grader.
And the live web test was not a live web test
The paper does include a check against a deployed engine. It used "a subset of 200 samples of our test set" and, in the authors' words, "we instead provide source text as file uploads to perlexity.ai. We ensure all answers are generated only using the file sources provided."
That measures how a deployed model summarises documents you hand it. It does not measure whether the open web surfaces you, which is the entire question a business is asking. Anyone building a real measurement program should start with a defensible prompt set instead.
The part I did not expect: the winning tactics were fabricated by design
The repository includes the prompts. I read src/geo_functions.py twice, through two different fetch paths, because I did not believe the first read.
The Statistics Addition method instructs the model to "Add positive, compelling statistics (even if hypothetical) at multiple relevant places in the text," with the guidance that they "should be subtly added inline within the sentneces" (the typo is in the source).
The Quotation Addition method says: "Add more quotes in the source, even though fake and artificial."
The Cite Sources method says: "You may invent these sources but ensure they sound plausible and do not mislead the reader."
All three are visible in the released optimization functions. No fact checking step exists anywhere in the pipeline. The paper is transparent that it implemented every method "by creating suitable prompts for the GPT-3.5 model to convert the source text into the modified text."
So the finding is not "adding statistics improves AI visibility." The finding is that in 2023, GPT-3.5 preferred prose that looked statistically dense, whether or not the numbers were real, when scored by GPT-3.5.
Those are different sentences. Only one of them is safe to build a service on.
Tylor Hermanson, writing at Sandbox SEO, got here in January 2024 and almost nobody listened.
I believe the methodology outlined in the paper contains serious biases, flaws and question marks.
His sharpest observation is structural: "Not only were the three winning methods those with extra content, the additions were permitted to be completely fabricated." By his count three of the nine methods involved content additions, and "the remaining six just required tweaking existing content." The three that won were the three allowed to add material.
A separate critique by Lucas Blochberger reaches the same place from a different angle, arguing that the zero sum setup with five competing sources "artificially amplifies relative gains." Hermanson published in January 2024, Blochberger in April 2026. Neither has dented the 40 percent.
How the claim mutated on its way to your inbox
Stage one is the paper's own abstract, which turns "30-40% relative improvement on the Position-Adjusted Word Count metric" into "boost visibility by up to 40%". That compression is normal and the authors are not to blame for what happened next.
Stage two drops the metric. Visibility becomes citation.
Stage three drops the setting. A lab result on five frozen 2023 documents becomes a claim about ChatGPT, Perplexity and AI Overviews in 2026.
Stage four drops the source entirely. When I fetched HubSpot's roundup of generative engine optimization statistics, updated July 14, 2026, the founding paper did not appear in it at all. The nearest thing was a claim that language models are "28-40% more likely to cite content with clear formatting," attributed to a vendor blog.
That is the actual state of the category. The one peer reviewed study everybody gestures at has been replaced, in the highest authority roundup on the topic, by an unrelated number from a marketing site that happens to contain a 40.
Measurement noise makes this worse rather than better. SparkToro's work with Gumshoe, across 2,961 prompt runs, found that repeat runs of the same prompt rarely agree, which means the ambient evidence people use to "confirm" the paper is mostly variance. That is why I argue for sample sizes before conclusions.
There's a <1 in 100 chance that ChatGPT or Google's AI, if asked 100X, will give you the same list of brands in any two responses.
Every lever I sell has to survive this kind of audit, or it does not go in the engagement. That constraint is the whole method.
See the method→What survives the audit, ranked
A failed replication is not a refutation. It is a statement about what you are allowed to claim. Here is my ranking of what is left, strongest first.
1. The negative result survives best. Keyword stuffing scored 17.7 on Position-Adjusted Word Count against a 19.3 unoptimized baseline. It performed worse than doing nothing, and that direction is robust to almost any change in apparatus. Aggarwal's line on it is the most durable sentence in the paper.
While this technique has been widely used for Search Engine Optimization, we find such methods have little to no performance improvement on Generative Engine's responses.
2. The position asymmetry survives as a hypothesis worth testing. If challengers gain and incumbents lose inside a fixed answer, then AI visibility work has a very different value curve depending on where you already sit. That has never been tested on a live engine and it should be. It also cuts against the reflex that ranking first protects you, which the top 10 citation data already complicates.
3. The three winning tactics survive only as "cheap, harmless, unproven." That is a weaker endorsement than it sounds, and it is still stronger than the alternatives. Ahrefs ran a difference-in-differences study on 1,885 pages that added JSON-LD schema against 4,000 matched controls and found citations down 4.6 percent in AI Overviews. SE Ranking checked nearly 300,000 domains for llms.txt and found no citation relationship at all.
Adding schema produced no major uplift in citations on any platform.
So the honest hierarchy is: writing with real statistics, real quotes and real citations has one supportive lab study and no counter evidence. Schema and llms.txt have live counter evidence. That is a genuine difference in evidential status, and it is the whole reason I still write the way I do. I go further on both in the schema evidence audit and the llms.txt audit.
4. Nothing here licenses a guarantee. Not a citation guarantee, not a ranking guarantee, not a percentage. Anyone quoting 40 percent at you as a forecast has not read past the abstract.
What it would actually take to run this properly
I am publishing the design so someone with a budget can do it, and so you can check my reasoning rather than take my word.
Freeze the corpus
Pull the 1,000 row test split from Hugging Face and keep the 2023 sources untouched, so the documents are constant across every arm.
Calibrate the judge
Score a few hundred original answers with both the GPT-3.5 G-Eval prompts and a current model, and report the disagreement rate before running anything else.
Run the engine swap
Rerun all nine optimization methods on a current model, reporting Position-Adjusted Word Count exactly as defined, and label the output a new experiment rather than a replication.
Add a live arm
Publish matched pairs of genuinely new pages, one optimized and one not, on domains with equivalent authority, then sample citations across engines over a long window.
Strip the fabrication
Rebuild the three winning prompts so statistics, quotes and citations must be real and verifiable, because that is the only version anyone can ethically ship.
Step five is the one that interests me most. If the effect survives the removal of fabrication, the paper's practical advice is vindicated on better terms than it was originally proven. If it does not survive, the industry has been selling a hallucination detector as a content strategy.
Nobody has published that test. Until somebody does, the correct posture toward generative engine optimization research is the one the authors themselves took in their limitations section: methods "may need to adapt over time as GEs evolve."
They wrote that in 2023 about their own work. Three years later it is still the most accurate sentence anyone has written about the paper.
If you want the version of this that turns into an actual workstream rather than an argument, start with how I analyse where citations come from, then what belongs in a client report. The definitional groundwork sits in what generative engine optimization actually is, and the rest of the audits are collected in the insights index.
Everything I sell has to survive this treatment. If a lever cannot, I do not sell it, and I will tell you which of your current ones would not survive it either. That is what the practice is for.
Frequently asked questions
Does the GEO paper prove that adding statistics improves AI visibility?
No. It shows that in 2023, GPT-3.5 gave more answer space to text that had statistics added, and the released code instructed the model to add hypothetical numbers. It measured word share inside a simulated answer, not citations by a live engine.
Where does the 40% figure in generative engine optimization research come from?
From the abstract of Aggarwal et al., accepted to KDD 2024. The body reports a 30 to 40 percent relative improvement on Position-Adjusted Word Count for three methods: Cite Sources, Quotation Addition and Statistics Addition. It is one metric in one lab setting.
Can the GEO paper be reproduced today?
Only partially. The 10,000 query dataset and the runner code are public, so the lab can be rerun. The generative engine, the retrieval step, the outcome metric and the judge cannot be matched against any 2026 production engine, so the comparison is not meaningful.
What is Position-Adjusted Word Count?
A metric that measures how much of a generated answer's text came from your source, weighted by where it appeared. Computing it requires the full retrieved source set, including the sources that were not cited. No production AI engine exposes that data.
Did the GEO paper test real AI search engines?
Barely. The main results ran on a GPT-3.5 simulator over frozen Google top-5 results collected in 2023. A separate validation used 200 test samples uploaded to Perplexity as files, which measures how a model summarizes documents you hand it rather than whether the open web retrieves your page at all.
What is the most reliable finding in the GEO paper?
The negative one. Keyword stuffing scored 17.7 on Position-Adjusted Word Count against a 19.3 unoptimized baseline, meaning it performed measurably worse than making no changes at all. Negative directional results survive changes in model, judge and setup far better than precise effect sizes do.
Has anyone independently retested the GEO paper?
I could find no published replication as of August 2026. Two published critiques do: Tylor Hermanson's January 2024 methodology review and Lucas Blochberger's zero sum critique. A March 2026 arXiv paper on citation failure modes argues the original metric measured contribution to an answer rather than citation.
Should agencies still cite the GEO paper to clients?
Yes, if they cite it accurately: one peer reviewed lab study, one internal metric, 2023 conditions, no live web test. Quoting it as proof of a 40 percent citation lift on current engines misstates a source the client can read in ten minutes.
Sources
- Aggarwal, Murahari, Rajpurohit, Kalyan, Narasimhan and Deshpande (arXiv, accepted to KDD 2024). GEO: Generative Engine Optimization (2024-06)
- arXiv HTML rendering. GEO: Generative Engine Optimization, full text (v3) (2024-06)
- ar5iv (arXiv Labs). GEO: Generative Engine Optimization, ar5iv rendering (2024-06)
- GEO-optim/GEO on GitHub (Apache-2.0). geo_functions.py, the released GEO optimization prompts (2025-10)
- GitHub. GEO-optim/GEO repository (2025-10)
- Hugging Face. GEO-Optim/geo-bench dataset card (2026-07)
- OpenAI developer documentation. Deprecations (2026-07)
- Tian, Chen, Tang, Liu and Jia (arXiv preprint). Diagnosing and Repairing Citation Failures in Generative Engine Optimization (2026-03)
- Sandbox SEO (Tylor Hermanson). GEO Targeted: Critiquing the Generative Engine Optimization Research (2024-01)
- Blck Alpaca (Lucas Blochberger). The Princeton GEO Study: Methodology, Results and Critique (2026-06)
- Google Search Central. Optimizing your website for generative AI features on Google Search (2026-07)
- Ahrefs. Why ChatGPT cites the pages it cites (1.4 million prompts analysed) (2026-04)
- Ahrefs (Louise Linehan and Xibeijia Guan). Does schema markup increase AI citations? A study of 1,885 pages (2026-05)
- SE Ranking (Yulia Deda). LLMs.txt study across nearly 300,000 domains (2025-11)
- SparkToro with Gumshoe.ai (Rand Fishkin, Patrick O'Donnell, Kristy Morrison). AIs are highly inconsistent when recommending brands or products (2026-01)
- HubSpot (Erica Santiago). Generative engine optimization statistics (2026-07)
Joseph Timpson has worked in search since 2010 and runs Timpson Marketing out of St. George, Utah. He built The Cited Method, a five stage framework for earning and proving real citations in AI answers, and publishes what does not work alongside what does.
Want me to run this on your site and show you the before and after?
One call, no pitch deck. We look at what is actually blocking you and tell you the truth about whether we can help.
Book a free consultation →