Can Every AI Crawler Actually Fetch This Site?
Reading robots.txt tells you what a site intends. It tells you nothing about what an AI agent actually receives. This is the procedure for the four layers underneath, and the honest limit on what any of it can prove.
- Blocking an AI crawler in Cloudflare's AI Crawl Control creates or updates a WAF custom rule on the zone, so the block lives in the firewall and never appears in robots.txt.
- On September 15, 2026 Cloudflare will block the Training and Agent categories by default for all new domains onboarding, on pages that display ads, while leaving Search allowed.
- OAI-SearchBot is the ChatGPT search gate, not GPTBot. Google-Extended controls Gemini training and does not affect Google Search inclusion. Blocking the wrong token costs visibility you meant to keep.
- You cannot prove access by spoofing a user agent with curl. Bot management scores IP reputation, ASN and TLS fingerprint too, so a curl 200 is a weak positive and a curl 403 is a strong negative.
- The only real proof is a log line: a request from a published, verified operator IP range that returned an HTTP status below 400 with a non-empty body.
The firewall rule you never wrote
An AI crawler access audit is finished when you hold proof of a successful fetch from a verified operator IP range, not when robots.txt looks clean. Cloudflare's own documentation states that when you block a crawler in AI Crawl Control, the system creates or updates a WAF custom rule on your zone to enforce that block. The decision lives in the firewall. robots.txt never mentions it.
That single implementation detail invalidates most of what gets published as AI crawlability advice. The industry standard audit is: open /robots.txt, search for GPTBot, find no Disallow, declare access. That check reads a file that has no authority over whether the request survives.
By August 2025, Cloudflare reported that over two and a half million websites had chosen to completely disallow AI training through its managed robots.txt feature or its managed rule blocking AI crawlers. Two mechanisms, one visible from the street, one not. If a site owner picked the second one, the robots.txt audit returns a clean bill of health on a site that answers every training crawler with a 403.
of AI crawling in July 2025 was for training rather than search or user actions
This matters more than it did a year ago because access is now the cheapest thing on the Cited Method checklist that is also the most certain. Nothing else in generative search has this property. Schema does not causally lift citations. llms.txt is not read by the engines that matter. But a 403 to OAI-SearchBot is a deterministic, reproducible, binary removal from ChatGPT's search answers, and OpenAI states it plainly: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
Five layers, and only one of them is public
A request from an AI agent can die in five places. Four of them are invisible to anyone outside the account.
| Layer | Who owns it | What the agent receives | Where it is visible |
|---|---|---|---|
| 1. robots.txt | SEO or dev | Full content. A compliant agent voluntarily skips | Public. Anyone can read it |
| 2. Origin server | Sysadmin, host, security plugin | 403, 406, or an empty 200 | Server config, .htaccess, plugin settings |
| 3. WAF custom rule | Security or CDN admin | 403 or 402, sometimes with a custom body | CDN dashboard only |
| 4. CDN managed ruleset | The vendor, by default | 403 or a block page | Vendor dashboard, and it changes without you |
| 5. Bot management or rate limit | Vendor heuristics | JS challenge, CAPTCHA, or 429 | Partly visible in bot analytics |
Here is the ranking I would defend, and I am labelling it as judgment rather than data because no published study measures the split. Ordered by how often each layer is the actual cause of a failed AI fetch, it goes 4, 5, 3, 2, 1. The vendor default is the most common blocker. robots.txt is the least common, and it is the only one anybody checks.
The reason is structural. Layers 2 and 3 require somebody to make a decision and type it. Layer 4 requires nobody to do anything at all.
Read the third figure next to the first. The Web Almanac's 2025 SEO chapter puts under five percent of sites in the camp that has an opinion about GPTBot written down, while the overwhelming majority of AI crawl volume is training traffic. Almost nobody is making this decision in robots.txt. That does not mean almost nobody is making it.
Three jobs, not one decision
Every major vendor has converged on the same split: a training crawler, a search or index crawler, and a user triggered fetcher. Treating "AI bots" as one switch is the single most expensive error in this whole area, because two of the three categories are how you get cited and only one of them is how you get scraped.
| Agent | Operator | Job | Documented cost of blocking it |
|---|---|---|---|
GPTBot | OpenAI | Trains foundation models | Signals content should not be used in training |
OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | Removed from ChatGPT search answers |
ChatGPT-User | OpenAI | User triggered fetch | OpenAI states robots.txt rules may not apply |
OAI-AdsBot | OpenAI | Validates ad landing pages | Not stated |
ClaudeBot | Anthropic | Model training | Excluded from future training datasets |
Claude-SearchBot | Anthropic | Indexes for Claude search | Reduced visibility and accuracy in search results |
Claude-User | Anthropic | User triggered fetch | Cannot retrieve your content for a user query |
PerplexityBot | Perplexity | Surfaces and links sites in results | Not surfaced in Perplexity results |
Perplexity-User | Perplexity | User triggered fetch | Documented to generally ignore robots.txt |
Googlebot | Search index | Removes Search, AI Overviews and AI Mode eligibility | |
Google-Extended | Gemini training and grounding control | No effect on Google Search inclusion | |
meta-externalagent | Meta | Training and direct indexing | Not stated |
meta-externalfetcher | Meta | Agentic navigation | Meta states it may bypass robots.txt rules |
CCBot | Common Crawl | Public corpus used downstream | Removed from the open crawl archive |
Every consequence in that right hand column is the operator's own published language, not my inference, taken from the crawler documentation published by OpenAI, Anthropic, Perplexity, Google, Meta and Common Crawl. That is the useful part. Each vendor now tells you on the record what you forfeit, which turns a vibes argument with a nervous client into a table.
prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results
Two rows deserve isolating because they are where practitioners lose visibility by accident.
OAI-SearchBot, not GPTBot, is the ChatGPT gate. A site that blocks GPTBot to protect its content from training has done nothing to its ChatGPT search presence. A site that blocks both, usually by pattern matching on the word "OpenAI" or copying a block list off a forum, has quietly removed itself from ChatGPT search answers.
Google-Extended is not a crawler and does not touch Search. Google's crawler overview is explicit: Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. It is a use-of-data token for Gemini training and grounding. Blocking it costs you Gemini grounding exposure and nothing in AI Overviews or AI Mode, which run on the ordinary Search index. Google's AI features documentation states there are no additional requirements to appear in AI Overviews or AI Mode beyond ordinary Search eligibility, and that a page must be indexed and eligible to be shown with a snippet.
The practical consequence of that last clause is under-audited. A stray nosnippet, a max-snippet:0, or a data-nosnippet wrapper around the section that actually answers the question is an AI visibility block that no crawler access audit will ever catch, because the fetch succeeds perfectly.
Layer one teardown: four robots.txt failures nobody checks
Before going below robots.txt, most files are wrong in ways that have nothing to do with AI. Google publishes the parsing rules and they contain four traps.
One. Tokens are case insensitive, paths are not. Google's spec states that the user-agent field name and its value are both case insensitive, while for disallow the field name is case insensitive but its value is case sensitive. So gptbot matches GPTBot fine, but Disallow: /Private/ does not block /private/. Third party robots parsers are less forgiving on the token than Google is, which is why writing the exact documented casing is still the right habit.
Two. The most specific group wins and the rest are ignored. Google's crawlers determine the correct group of rules by finding the group with the most specific user agent that matches the crawler. The moment you write a User-agent: GPTBot block, GPTBot stops reading your User-agent: * block entirely. Every Allow and Disallow you carefully wrote in the wildcard group vanishes for that agent. This is the most common way a well intentioned AI section breaks an otherwise correct file.
Three. There is a hard parse ceiling. Google enforces a robots.txt file size limit of 500 kibibytes, and content after the maximum file size is ignored. Enterprise files padded with thousands of legacy disallows can push the AI section past the cliff.
Four. A 5xx on robots.txt is worse than a 404. Google treats all 4xx errors except 429 as if a valid robots.txt file did not exist, meaning it assumes there are no crawl restrictions. A 5xx is the opposite: for the first 12 hours Google stops crawling the site while it keeps retrying the file. An origin that throws 500s on /robots.txt under load is throttling itself.
One more, carried over from classic SEO and still in Google's noindex documentation: a Disallow plus a noindex cancels the noindex, because a blocked crawler never sees the meta tag. If you want a page out of AI answers, blocking the fetch is the wrong instrument.
Layers two through five: the config you cannot see from the street
Layer 2, the origin. Security plugins, mod_security rule sets, and managed WordPress hosts ship user agent deny lists. These are usually the residue of a scraping incident two years ago. They return 403 or, worse, a 200 with an empty body, which every automated crawlability tool scores as a pass.
Layer 3, the WAF custom rule. This is where an explicit decision lands. It is also where implicit ones land, because Cloudflare's AI Crawl Control writes a WAF custom rule when you toggle a crawler to block. The block response is configurable to 403 Forbidden or 402 Payment Required with a custom body. If you have ever wondered why a client's site returns 402 to an AI agent, that is the mechanism, and it was probably two clicks in a product UI rather than a firewall project.
Layer 4, the vendor default. This is the layer that produces the case in the headline. On July 1, 2025, Cloudflare announced that every new domain signing up is asked whether it wants to allow AI crawlers, with the default set to block. That is a live, load bearing default. A site launched on a fresh zone in the last year can be blocking training crawlers without a single human having chosen it.
And the default is moving again. From Cloudflare's July 1, 2026 post:
On September 15, 2026, we'll be setting new defaults for each of these three classifications. For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default.
Read the second half carefully, because it is a genuine improvement and the coverage has mostly missed it. Cloudflare is splitting AI traffic into Search, Agent and Training, and it is leaving Search allowed. Its stated reasoning is that Search is the behavior that most naturally funnels back visitors. For anyone whose job is AI visibility rather than content licensing, that is the vendor moving toward the correct default. It also means the audit question changes from "are AI bots blocked" to "which of the three categories is blocked, on which URLs", since the new default is scoped to pages that display ads.
Layer 5, bot management and rate limits. The same post contains the sentence that should worry anyone who audited a site last quarter and filed it as clean: previously, all Verified bots were allowed by default, which was reflected in Cloudflare's basic Bot Fight Mode offering and in its rule templates for Enterprise Bot Management customers. Now the Verified label makes a bot allowable with its relevant category, so the allowed category determines what reaches the site. Cloudflare's verified bots documentation says the same thing from the other side: historically, Verified bots have been excluded in default bot configurations across all plans, and verification does not automatically grant access.
Bot Fight Mode itself is a free product that issues computationally expensive challenges to traffic matching known bot patterns, and Cloudflare warns in its own docs that these products may challenge API or mobile app traffic. A challenge is not a 403. It is a 503 with a JavaScript payload, and since no major AI crawler executes JavaScript, a challenge is a total block wearing a friendly status code.
That last point deserves its own line. Vercel and MERJ's log study found GPTBot fetched JavaScript files in 11.50 percent of requests and ClaudeBot in 23.84 percent, but neither executed them. That study is from December 2024 and nobody has replicated it at scale since, so treat the specific percentages as dated. The architectural claim, that a JS challenge is an impassable wall for a non-rendering agent, has not been contradicted by anything published since.
ACCESS is stage one of the Cited Method for a reason. Everything downstream, the measurement, the mapping, the earning, is arithmetic on zero if the fetch never lands.
See how ACCESS fits the method→You cannot prove access by spoofing a user agent
This is the part the tool pages get wrong, and it is the reason a "free AI crawler checker" that returns green is close to worthless.
The standard test is to send a request with the agent's user agent string and read the status code:
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://example.com/page
That request tests exactly one variable: whether a string match on the user agent header triggers a block. Layers 2 and 3 usually are string matches, so this test genuinely catches them. Layers 4 and 5 are not string matches. Bot management scores source IP reputation, originating ASN, TLS handshake fingerprint, HTTP/2 frame ordering and request cadence. Your laptop shares none of those with OpenAI's infrastructure, and curl's TLS fingerprint matches neither Chrome nor GPTBot.
So the logic is asymmetric, and the asymmetry is the whole methodology:
| Result: blocked | Result: 200 OK | |
|---|---|---|
| What it proves | A user agent string block exists. Real and actionable | Nothing about layers 4 and 5. Your IP is not their IP |
| Confidence | High. Strong negative | Low. Weak positive |
| Correct next action | Find and remove the rule | Do not close the ticket. Go to the logs |
Spoofing can disprove access. It cannot prove it. Anyone selling you a green checkmark from a headless request is selling you a false negative machine.
Proof lives on the log side, and every serious operator now publishes what you need to build it. Google documents two methods, a reverse DNS lookup on the accessing IP that must resolve to googlebot.com, google.com or googleusercontent.com and then forward resolve back to the same IP, plus five published CIDR files including common-crawlers.json and user-triggered-fetchers.json. OpenAI publishes one JSON per bot at openai.com/gptbot.json, searchbot.json, chatgpt-user.json and adsbot.json, and recommends allowing requests from those published ranges rather than trusting the header. Anthropic publishes claude.com/crawling/bots.json. Perplexity publishes perplexity.com/perplexitybot.json and perplexity-user.json. Common Crawl publishes index.commoncrawl.org/ccbot.json and reverse DNS under crawl.commoncrawl.org, and warns on its own page that crawlers falsely identify themselves as CCBot.
Meta publishes neither IP ranges nor an ASN in its crawler documentation. It recommends allow listing "either the user agent strings or the IP addresses used by the crawler" without saying what those addresses are. That is a documentation gap worth naming rather than working around.
Why IP verification is not optional: in August 2025 Cloudflare published an investigation into Perplexity's undeclared crawlers finding Perplexity repeatedly modifying its user agent and changing its source ASNs to hide crawling activity, at three to six million daily requests from the stealth crawler against twenty to twenty five million from the declared one. Cloudflare de-listed Perplexity as a verified bot. Whatever you think of the dispute, the operational lesson stands in both directions: a user agent header is a claim, not an identity, so both your block lists and your allow lists are guessing until you check the IP.
Follow the rules. That means checking for and respecting website signals like robots.txt, staying within rate limits, and never bypassing security protections.
The same investigation contains the best experimental design published in this area, and it is worth stealing. Cloudflare created multiple brand new domains, newly purchased, not indexed by any search engine and not publicly discoverable in any way, then served a restrictive robots.txt and watched what arrived. That is a clean room. If you run agency infrastructure, a permanently maintained clean room domain with known robots.txt states is the cheapest possible standing instrument for AI crawler log analysis.
The runnable procedure
Fetch robots.txt as the agent, not as yourself
Some WAF rules block /robots.txt itself. Request it with each target user agent and confirm a 200 and a non-empty body. A blocked robots.txt means every compliant agent falls back to assuming no restrictions, which is a different problem than the one you were looking for.
Parse the file the way Google does
Confirm each named agent group, remembering the most specific matching group wins and the wildcard group is then ignored for that agent. Check path casing. Check total file size against the 500 KiB ceiling.
Separate the three jobs
For each operator, record the training token, the search token and the user fetch token independently. A single allow or deny verdict per vendor is not an answer.
Spoof-test a real content URL, not the homepage
Homepages are often exempted from rules that hit deep paths. Request a money page and a blog post. Record status, byte count and whether the answer text is present in the raw HTML, since no major AI crawler executes JavaScript.
Treat any challenge as a block
A 503 with a JS challenge, a CAPTCHA interstitial or a 429 is a failed fetch for a non-rendering agent. Do not score it as a pass because it is not a 403.
Open the CDN dashboard and read the actual rules
Custom WAF rules, managed rulesets, bot fight mode state, rate limits, and any AI crawler product toggles. This is the step the spoof test cannot replace and the step most audits skip because it needs client access.
Verify from the logs
Pull 30 days of access logs or CDN analytics. For each agent, confirm at least one request from a published operator IP range that returned a status below 400 with a non-empty body. That log line is the deliverable. Everything before it is inference.
Re-check the vendor defaults on a calendar
Cloudflare changed its new-domain default on July 1, 2025 and is changing the category defaults on September 15, 2026. An access audit has a shelf life measured in months.
If you are running this inside a client engagement rather than as a one-off, the log verification step is what belongs in the scope of work and the recurring re-check is what belongs in the monthly report. Access is the only part of generative search work with a binary, demonstrable pass state, which makes it the easiest thing you will ever have to prove to a client.
What this audit cannot tell you
The strongest argument for doing it anyway is not upside, it is asymmetry. Access failures are total and silent. A blocked agent does not rank badly, it does not exist. And unlike almost everything else in this field, the fix is deterministic and the verification is a log line rather than a sampled estimate, which is a rare thing when the measurement layer is this noisy.
The web is not disappearing. Humans are simply becoming the minority of those who read it.
One caveat on my own framing. I have been describing the risk as accidental blocking, which assumes you want the agents in. That assumption is not universal and it is not obviously right. Cloudflare's crawl to click analysis measured crawl to refer ratios in 2025 as high as 500,000 to 1 for Anthropic and 3,700 to 1 for OpenAI's GPTBot, against 3 to 1 through 30 to 1 for Google.
With OpenAI, it's 750 times more difficult to get traffic than it was with the Google of old. With Anthropic, it's 30,000 times more difficult.
A publisher who monetises page views has a defensible reason to block training crawlers and keep search crawlers. That is precisely the split Cloudflare is now productising, and it is the right shape for the decision. The failure mode this post is about is not choosing to block. It is discovering, six months into a generative engine optimisation engagement, that a choice was made for you by a default, on a layer you never opened, on a site whose robots.txt has been clean the whole time.
One final data point, filed with its uncertainty attached. Cloudflare Radar exposes an AI bot response status endpoint, and a compiled reading of that endpoint puts 403 responses at 8.56 percent of AI bot requests network wide across April to June 2026. I could not fetch Radar directly to confirm it, since the page returns 403 to automated requests, which is a joke I am choosing not to make. Treat the number as directional rather than settled. Directionally, roughly one in twelve AI crawler requests on the largest network with public data is being refused, and almost none of those refusals are written in a robots.txt file.
Frequently asked questions
Does a clean robots.txt mean AI crawlers can access my site?
No. Cloudflare's documentation states that blocking a crawler in AI Crawl Control creates or updates a WAF custom rule on your zone. That block is enforced at the firewall and is invisible in robots.txt. A permissive robots.txt is a statement of intent, not evidence that a fetch succeeds.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot crawls content to train OpenAI's foundation models. OAI-SearchBot surfaces sites in ChatGPT's search feature. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. If you want training protection without losing ChatGPT visibility, block GPTBot and allow OAI-SearchBot.
Does blocking Google-Extended remove me from AI Overviews?
No. Google's crawler documentation states that Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal. Google-Extended governs Gemini model training and grounding. AI Overviews and AI Mode run on the ordinary Search index, so Googlebot access and snippet eligibility are what matter there.
Can I test AI crawler access with curl?
Partly. A curl request with the agent's user agent string reliably detects user agent based blocks at the origin and in WAF custom rules. It cannot test bot management, which also scores IP reputation, ASN and TLS fingerprint. A curl block is strong evidence. A curl 200 is weak evidence and needs log confirmation.
How do I verify a request really came from an AI crawler?
Check the source IP against the operator's published ranges. OpenAI publishes one JSON per bot, Anthropic publishes claude.com/crawling/bots.json, Perplexity publishes two files, and Common Crawl publishes ccbot.json. Google supports both published CIDR files and reverse DNS resolving to googlebot.com, google.com or googleusercontent.com.
Do user triggered AI fetchers obey robots.txt?
Often not, and the operators say so. Perplexity documents that Perplexity-User generally ignores robots.txt rules because a user requested the fetch. OpenAI states that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. Meta states meta-externalfetcher may bypass robots.txt rules.
Is Cloudflare Bot Fight Mode blocking AI crawlers?
It can. Bot Fight Mode issues computationally expensive JavaScript challenges to traffic matching known bot patterns, and Cloudflare warns it may challenge legitimate API and app traffic. Since no major AI crawler executes JavaScript, a challenge is a complete block even though the status code is not a 403.
How often should an AI crawler access audit be repeated?
At least quarterly, and immediately after any CDN or security change. Cloudflare set new domains to block AI crawlers by default on July 1, 2025, and announced new category defaults taking effect September 15, 2026. Vendor defaults move without customer action, so a clean audit expires.
Will fixing crawler access get my site cited by AI?
It makes citation possible, not likely. Google states a page must be indexed and eligible to be shown with a snippet, and that there are no additional requirements for AI Overviews or AI Mode. Access removes a total, silent failure. It does not improve your position in retrieval ranking.
Sources
- Cloudflare AI Crawl Control documentation. Manage AI crawlers (2026-07)
- Cloudflare AI Crawl Control documentation. Track robots.txt compliance (2026-07)
- Cloudflare. Your site, your rules: new AI traffic options for all customers (2026-07-01)
- Cloudflare (press release). Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large (2025-07-01)
- Cloudflare (Reid Tatoris). Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives (2025-08-04)
- Cloudflare documentation. Verified bots (2026-07)
- Cloudflare documentation. Bot Fight Mode (2026-07)
- Cloudflare (Joao Tome). The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (2025-08-29)
- Cloudflare Radar. The 2025 Cloudflare Radar Year in Review (2025-12)
- Cloudflare (Matthew Prince). Content Independence Day: no AI crawl without compensation! (2025-07-01)
- OpenAI developer documentation. OpenAI bots (2026-07)
- Perplexity AI documentation. Perplexity Crawlers (2026-07)
- Anthropic support documentation. Does Anthropic crawl data from the web, and how can site owners block the crawler? (2026-04-07)
- Google Search Central. Google's common crawlers (2026-07-14)
- Google Search Central. Verifying Googlebot and other Google crawlers (2026-07)
- Google crawling infrastructure documentation. How Google interprets the robots.txt specification (2026-07)
- Google Search Central. Overview of Google crawlers and fetchers (2026-07)
- Google Search Central. AI features and your website (2025-12-10)
- Google Search Central. Block indexing with noindex (2025-12-10)
- Meta developer documentation. Meta web crawlers (2026-07)
- Common Crawl Foundation. CCBot (2026-07)
- Vercel with MERJ. The rise of the AI crawler (2024-12-17)
- HTTP Archive. Web Almanac 2025, SEO chapter (2026-01-15)
- iPullRank (Michael King). Machine Media (2026-05-26)
- TechnologyChecker (David Thomson), compiling Cloudflare Radar data. We Analyzed robots.txt Across Cloudflare's Network (2026-07-02)
- Cloudflare documentation. AI Crawl Control overview (2026-07)
Joseph Timpson has worked in search since 2010 and runs Timpson Marketing out of St. George, Utah. He built The Cited Method, a five stage framework for earning and proving real citations in AI answers, and publishes what does not work alongside what does.
Want me to run this on your site and show you the before and after?
One call, no pitch deck. We look at what is actually blocking you and tell you the truth about whether we can help.
Book a free consultation →