# Do You Block AI Crawlers by Accident? CDN and Plugin Traps > Most sites that block AI crawlers never decided to. A CDN default, a security rule aimed at training bots, a plugin or a JavaScript-only theme stops OAI-SearchBot, PerplexityBot or Googlebot before a page is read. Check each layer in order, verify bots by IP in your edge and server logs, and allow search bots explicitly. By Othmane Khadri, founder of Earleads. Published 2026-10-20. Canonical: https://www.earleads.com/blog/blocking-ai-crawlers-by-accident/ ## Why sites block AI crawlers without deciding to Most companies that block AI crawlers never made that call. Nobody in marketing wrote "remove us from ChatGPT" on a roadmap. The block came from a security default, a vendor update, a rule meant for one bot that caught another, or a site theme that hides its text from anything that doesn't run JavaScript. That's why "check your robots.txt" isn't enough. A crawler request passes through several layers before it reads a word of your page, and any one of them can stop it. Your robots.txt can say yes while your CDN says no, and the CDN wins. At Earleads, crawler access is the first thing we check in every audit, on day 1, and the result goes to the client the same day. If AI crawlers can't reach the site, nothing else we measure is worth much. The worst case we've dealt with wasn't robots.txt at all. A site theme rendered every word through JavaScript, so AI crawlers saw code and no text, and the gated whitepapers were invisible on top of that. The fix was server-side rendering plus explicitly allowing the AI search bots. ![The layers an AI crawler request passes through before it reads a page, robots.txt, CDN bot settings, firewall and plugins, JavaScript rendering and forms, with a block possible at each one before the readable text](/blog/blocking-ai-crawlers-by-accident/diagram-1.svg) *A crawler has to get through every layer. Robots.txt is only the first one, and often not the one doing the blocking.* If you want to block AI crawlers on purpose, that's a legitimate choice, and the steps below show you where each control lives. This guide is for the other case, the company that wants to be cited and can't work out why it isn't. ## Which AI crawlers matter for visibility AI companies run several bots, and they don't do the same job. Blocking a training bot is a content decision. Blocking a search bot or a user-triggered fetcher removes you from answers. Cloudflare now sorts AI traffic into three purposes. Search is "any behavior that collects or indexes your content, so it can answer questions about it later." Training is "a crawler taking your content to train or fine-tune a model." Agent is "automated behavior that is acting, usually in real time, on a person's behalf" ([Cloudflare](https://blog.cloudflare.com/content-independence-day-ai-options/)). | Bot | Company | Purpose | Respects robots.txt | Blocking it means | |---|---|---|---|---| | OAI-SearchBot | OpenAI | Search | Yes | Not shown in ChatGPT search answers | | GPTBot | OpenAI | Training | Yes | Content not used to train OpenAI models | | ChatGPT-User | OpenAI | Agent (user-triggered) | "may not apply" | ChatGPT can't open your page when a user asks | | PerplexityBot | Perplexity | Search | Yes | Not surfaced in Perplexity results | | Perplexity-User | Perplexity | Agent (user-triggered) | "generally ignores" | Perplexity can't visit the page for a user | | Googlebot | Google | Search, also feeds AI Overviews | Yes | Gone from Search and AI Overviews | | Google-Extended | Google | Training and Gemini grounding (a token, not a bot) | Yes | No effect on Google Search | The sources are the companies' own pages. OpenAI says OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features," and for ChatGPT-User, "because these actions are initiated by a user, robots.txt rules may not apply" ([OpenAI](https://developers.openai.com/api/docs/bots)). Perplexity says PerplexityBot is "designed to surface and link websites in search results on Perplexity" and "is not used to crawl content for AI foundation models," while Perplexity-User "generally ignores robots.txt rules" ([Perplexity](https://docs.perplexity.ai/guides/bots)). Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" and "doesn't have a separate HTTP request user agent string" ([Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)). Two consequences follow. First, the user-triggered fetchers aren't governed by robots.txt the way crawlers are, so if they're blocked, it's almost always happening at the CDN or firewall. Second, Google-Extended is a robots.txt token only, which means a firewall rule can't target it. Any firewall rule aimed at "Google's AI" hits Googlebot itself. For the basics of OpenAI's three bots and a robots.txt that allows search while opting out of training, see [how to get ChatGPT to recommend your business](/blog/how-to-get-chatgpt-to-recommend-your-business/). The rest of this guide is about everything after robots.txt. ## Step 1: Read robots.txt the way the bots read it Start with the file, but read it like a crawler, not like a person skimming for "GPTBot." - **Open the live file.** Go to `yourdomain.com/robots.txt` on every hostname that serves content, including `www`, the apex domain, the blog subdomain, docs and the help center. Each host has its own file. - **Look for wildcards.** A `User-agent: *` block with `Disallow: /` applies to every bot that doesn't have its own group. Staging rules that leaked into production look exactly like this. - **Check group order and matching.** A bot follows the most specific group that names it. If OAI-SearchBot has no group of its own, it falls back to `*`. - **Look for generated rules.** Many CMSs and SEO plugins write robots.txt for you. The file you see in your CMS settings isn't always the one being served. Fetch the live URL. Then confirm Google sees the same thing. Search Console's robots.txt report shows the version Google last fetched, and whether it could fetch it at all. OpenAI and Perplexity both say robots.txt changes take about 24 hours to reach their systems, so don't re-test an edit an hour later and conclude it failed. ## Step 2: Check your CDN's AI bot settings This is where most accidental blocks live in 2026, because CDNs have been adding AI controls fast and changing their defaults. Cloudflare is the clearest example, and it documents its own behaviour. It said that "on September 15, 2026, we'll be setting new defaults for each of these three classifications" ([Cloudflare](https://blog.cloudflare.com/content-independence-day-ai-options/)). For new domains, "bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed" ([Cloudflare changelog](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/)). Read that carefully. Agent includes "chat fetch bots (e.g., ChatGPT-User)." So a new site that shows ads, on default settings, can stop ChatGPT from opening its pages when a user asks about it, while robots.txt says nothing of the kind. Where to look on Cloudflare. 1. **AI Crawl Control.** Formerly AI Audit, it's "available on all plans," shows which AI services request your pages and lets you allow or block each crawler ([Cloudflare](https://developers.cloudflare.com/ai-crawl-control/)). Check the per-crawler status for OAI-SearchBot, ChatGPT-User, PerplexityBot and Perplexity-User. 2. **Block AI bots.** The older setting has three options, "Block (on all pages)", "Block on pages with ads" and "Allow (do not block)" ([Cloudflare](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/)). Find out which one is on and who chose it. 3. **Bot Fight Mode.** Cloudflare says "you cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules" ([Cloudflare](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/)). If it's challenging a bot you want, an allow rule elsewhere won't save it. Exceptions need Super Bot Fight Mode. 4. **Managed robots.txt.** Cloudflare can add AI directives to your robots.txt for you. If the live file has rules nobody on your team wrote, this is a likely source. Other CDNs, hosts and bot management vendors have their own versions of these switches, often under names like bot protection, AI scrapers or crawler control. The question to ask your provider is the same everywhere. Which AI bots do you challenge or block on our domains today, under which setting, and since when? ## Step 3: Watch for multi-use crawler traps The most expensive accidental block is a rule aimed at training that also hits search. Some of the biggest crawlers do both jobs. Cloudflare calls them mixed-use, "a single crawler serving both search and AI training," and names Applebot, Bingbot and Googlebot ([Cloudflare](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/)). Its July post spelled out the consequence. "Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training" ([Cloudflare](https://blog.cloudflare.com/content-independence-day-ai-options/)). So a security team that blocks "AI training" to protect content can, in one click, block the crawler behind Google Search and AI Overviews, and the one behind Bing. Google says pages appear in AI Overviews and AI Mode only if they're "indexed and eligible to be shown in Google Search with a snippet" ([Google](https://developers.google.com/search/docs/appearance/ai-features)). Block Googlebot and both go. ![Two paths for a site that wants to opt out of AI training on Cloudflare. Blocking Training also blocks mixed-use crawlers like Googlebot, Bingbot and Applebot, so search visibility goes too. Disallow AI Training publishes a no-training preference in robots.txt and keeps accountable mixed-use crawlers allowed for search](/blog/blocking-ai-crawlers-by-accident/diagram-2.svg) *Opting out of training is fine. Doing it with a block that also stops mixed-use crawlers takes search with it.* Cloudflare added a setting for exactly this. "Disallow AI Training" "publishes the applicable no-training preference in robots.txt. Accountable mixed-use crawlers remain allowed for search. Every other training crawler is blocked" ([Cloudflare](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/)). If your company wants to opt out of training, that's the shape of the decision. Say no to training, keep search open. The same trap exists outside Cloudflare. Any firewall rule that matches "Google" or "Bing" in the user agent to stop AI use catches the search crawler, because for Google there's no separate AI crawler to match. Use Google-Extended in robots.txt for Gemini, and `nosnippet`, `data-nosnippet`, `max-snippet` or `noindex` if you want to limit how a page appears in Google's AI features. This happens a lot in practice. BuzzStream's study of the top 50 news sites in the US and the top 50 in the UK found 62% blocking GPTBot and 49% blocking OAI-SearchBot ([BuzzStream](https://www.buzzstream.com/blog/publishers-block-ai-study/)). Some of that is deliberate. Some of it is a training decision that spilled over into search. ## Step 4: Audit plugins, site builder toggles and firewall rules Below the CDN sit more switches, usually installed by someone who has since left. **Security and SEO plugins.** On CMSs like WordPress, security, caching and SEO plugins can block AI bots, sometimes all of them regardless of the plugin's own settings page. List every plugin that touches robots.txt, headers, bots or rate limits, and test with it on and off in staging. **Site builder toggles.** Hosted builders often have a single checkbox. Read exactly what it blocks. Squarespace's "Block known artificial intelligence crawlers" under Settings, then Crawlers, is unchecked by default because the company says it doesn't "want to potentially impact your site's traffic," and its published list covers training bots such as GPTBot, ClaudeBot and Google-Extended without listing OAI-SearchBot, ChatGPT-User or PerplexityBot ([Squarespace](https://support.squarespace.com/hc/en-us/articles/360022347072-Request-that-AI-models-exclude-your-site)). That's a well-scoped toggle. Not every tool's is, so check the list behind yours. **Firewall and WAF rules.** Look for custom rules that block or challenge by user agent, country, ASN or request rate. AI search bots crawl from cloud data centres, so "block hosting providers" or "challenge datacenter ASNs" rules catch them. Rate limits tuned for human browsing can throttle a crawler mid-site. **Origin server rules.** `.htaccess`, nginx configs and middleware sometimes carry user agent blocklists copied from a forum years ago. Search the configs for bot names. Every finding here should become a ticket a developer can ship in under a day, ordered by the revenue of the pages it affects. A list of twenty "consider reviewing" items doesn't get shipped. ## Step 5: Make sure crawlers can read the text Getting through the door doesn't help if there's nothing to read once inside. Vercel and MERJ studied crawler traffic and found "none of the major AI crawlers currently render JavaScript," naming OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot and PerplexityBot among them ([Vercel](https://vercel.com/blog/the-rise-of-the-ai-crawler)). Google gives the same advice for its own index. "Server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript" ([Google](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics)). To test it, fetch the raw HTML of a key page, without running scripts, and look for your actual sentences. `curl -s https://yourdomain.com/pricing/ | grep "a sentence from the page"` is enough. If the HTML is a shell with a script tag and an empty `div`, that's what AI crawlers read. The usual causes are these. - Single-page apps that render all content in the browser - Themes or page builders that inject text through scripts - Pricing, specs and comparison tables loaded from an API after page load - Tabs and accordions whose content is fetched on click - Cookie or consent walls that replace the page until a click The fix is server-side rendering, static generation or pre-rendering for the pages that matter. Start with the ones that convert, not the ones with the most traffic. That usually means pricing, product and comparison pages, and the docs buyers read before they decide. ## Step 6: Take buyer content out from behind forms Content behind a form is invisible to AI. No crawler fills in a lead form, and no assistant will cite a whitepaper it can't open. This hits B2B companies hardest, because their most specific content, the benchmarks, the compliance guides, the implementation detail, is often the gated asset. In regulated markets it's worse, since the compliance content buyers check is exactly the content that sits behind a form. You don't have to ungate everything. Publish a plain-text page for each gated asset that states its key findings, its method and its date, with the form below for the full version. The page gets read and cited, and the form still captures the people who want more. The same applies to logged-in help centers and docs. If buyers ask AI how your product handles something, the answer has to be on a public page. ## Step 7: Check the logs at the edge and at the origin Settings tell you what should happen. Logs tell you what did. Don't test by changing your user agent on a laptop. That proves nothing, because CDNs and bot tools check the real IP behind a request, not only the name it claims. Your laptop pretending to be OAI-SearchBot can be blocked when the real bot isn't, or let through when the real bot is challenged. ![How to check AI crawler access in logs. A request claims a user agent like OAI-SearchBot or ChatGPT-User. Match its IP against the company's published JSON range. Then compare the CDN edge log, where blocks and challenges appear, with the origin server log, where only requests that got through appear](/blog/blocking-ai-crawlers-by-accident/diagram-3.svg) *Verify the bot by IP, then compare the edge with the origin. A block at the edge never shows up in your server logs.* Do it in three passes. 1. **Filter by user agent.** Look for OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Googlebot and Bingbot. ChatGPT-User and Perplexity-User matter most, because they appear when a real person asked an assistant about you and it tried to open your page. 2. **Verify by IP.** OpenAI publishes its ranges at `openai.com/searchbot.json`, `openai.com/gptbot.json` and `openai.com/chatgpt-user.json` ([OpenAI](https://developers.openai.com/api/docs/bots)). Perplexity publishes `perplexity.com/perplexitybot.json` and `perplexity.com/perplexity-user.json` ([Perplexity](https://docs.perplexity.ai/guides/bots)). A request that claims a bot name from outside those ranges is someone else. 3. **Compare edge and origin.** Your CDN's security events show blocks, challenges and rate limits. Your origin server logs only show what got through. A verified bot that appears at the edge with a block or challenge, and never at the origin, is your answer. A bot that appears at neither may be blocked by robots.txt, or may simply not have visited yet. Then check the status codes for verified bots that did reach the origin. A 403, a 429 or a 200 that returns a challenge page all mean the same thing to an answer engine. It didn't get your content. ## Fixing it without opening the door to everything Allowing AI search doesn't mean dropping your security. The fix is to make the decision explicit, per purpose. - **Search bots.** Allow OAI-SearchBot, PerplexityBot, Googlebot and Bingbot by name in robots.txt, and make sure your CDN and firewall let the verified versions through. - **User-triggered fetchers.** Allow ChatGPT-User and Perplexity-User on the pages buyers ask about. They show up because a person asked about you. - **Training bots.** Decide as a business, with legal and content owners in the room. Block GPTBot, ClaudeBot or Google-Extended if you choose to, using the control that doesn't also stop search, like Cloudflare's Disallow AI Training rather than a block on mixed-use crawlers. - **Everything else.** Keep challenging unverified bots and scrapers. Verification by IP is what lets you be strict and open at the same time. Write the decision down, with an owner, and re-check after every CDN, plugin or theme change. Defaults keep moving, and the next one won't ask you either. ## What changes for B2B and B2C companies **B2B.** The pages that need to be readable are the ones a buying committee checks, such as pricing, security and compliance, integrations, comparisons and docs. These are also the pages most likely to be gated, rendered through scripts or sitting on a separate docs subdomain with its own robots.txt and its own CDN settings. Enterprise security teams tend to run the strictest bot rules, so marketing and security need one shared decision on AI search bots. When a site's crawler access changes, note the date, so you can read changes in AI referrals and branded search against it, as we cover in [AI search analytics](/blog/ai-search-analytics/). **B2C.** Shoppers ask assistants which product, store or app to pick, and assistants need product pages, prices and stock they can read. Store themes and apps that load prices and variants through scripts are the usual culprits, along with bot protection that treats any non-browser as a scraper. If you sell online, check product and collection pages first, and see our guide on whether [your store is visible to AI shopping agents](/blog/is-your-store-visible-to-ai-shopping-agents/). ## A 30 minute AI crawler check - Open the live robots.txt on every content hostname and find the group each AI search bot follows - Check Search Console's robots.txt report for the version Google fetched - List your CDN's AI and bot settings, with who turned each one on and when - Confirm no "block training" setting also blocks Googlebot, Bingbot or Applebot - List plugins, builder toggles, firewall rules and server configs that touch bots - Fetch raw HTML for pricing, product and comparison pages and find your own sentences in it - Find gated or logged-in content buyers ask about, and plan public summary pages - Pull a week of edge and origin logs, verify bots by IP and compare the two Fixing access is the floor, not the strategy. Once crawlers can read you, what gets you cited is content that answers the questions buyers ask, covered in [how to get cited by AI](/blog/how-to-get-cited-by-ai/), and a measurement plan like the one in [what an AI visibility audit should show you](/blog/what-an-ai-visibility-audit-should-show-you/). Every [Earleads](/) audit starts with the crawler check on day 1, and you can [request one here](/#audit). ## Frequently asked questions ### How do I know if my site is blocking AI crawlers? Check each layer a request passes. Read robots.txt for OAI-SearchBot, PerplexityBot and Googlebot, then your CDN's bot and AI settings, firewall rules and any security plugin. Then look at your logs for requests from those bots, verified against the IP ranges each company publishes. If the edge shows blocks or the origin shows nothing, something is stopping them. ### Does Cloudflare block AI crawlers by default? For new domains, partly. Cloudflare says that from September 15, 2026, bots classified as Training or Agent are blocked on pages that display ads, while Search stays allowed. It also says that when a site blocks Training, multi-purpose crawlers such as Googlebot, Applebot and Bingbot get blocked too. Check your own settings rather than assuming. ### Should I block GPTBot but allow OAI-SearchBot? That's a business decision, and the two are separate. OpenAI says OAI-SearchBot surfaces websites in ChatGPT's search features, while GPTBot collects content that may be used for training. You can block one and allow the other in robots.txt. If you want to be cited in ChatGPT search answers, OAI-SearchBot has to get through every layer, not just robots.txt. ### Can I test AI crawler access by changing my user agent? Not reliably. Many CDNs and bot tools check where a request comes from, not only the name it gives. A laptop pretending to be OAI-SearchBot can be blocked when the real bot isn't, or let through when the real bot is challenged. Use your logs instead, and match requests against the IP ranges OpenAI and Perplexity publish. ### Do AI crawlers run JavaScript? Many don't. Vercel and MERJ found that none of the major AI crawlers they studied rendered JavaScript, including OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot. If your text only appears after scripts run, those crawlers fetch a page with code and no words. Server-side rendering or pre-rendering puts the text in the HTML they actually read. ### Does blocking Google-Extended remove us from AI Overviews? No. Google says Google-Extended does not affect inclusion in Google Search and isn't a ranking signal. It controls use of your content for Gemini training and grounding. AI Overviews and AI Mode use pages indexed by Googlebot, so blocking Googlebot, at robots.txt or at your CDN, is what removes you from them.