--- title: How to Block AI Bots in robots.txt: GPTBot, ClaudeBot & More (2026) | Mersel AI site: Mersel AI site_url: https://mersel.ai description: A technical guide for B2B and ecommerce brands on configuring robots.txt and CDN settings to block AI training crawlers while maintaining visibility in AI search engines like ChatGPT and Perplexity. page_type: blog url: https://mersel.ai/blog/how-to-block-or-allow-ai-bots-on-your-website canonical_url: https://mersel.ai/blog/how-to-block-or-allow-ai-bots-on-your-website language: en author: Mersel AI breadcrumb: Home > Blog > How to Block or Allow AI Bots date_modified: 2024-05-22 --- > Strategic AI bot management is critical as AI-referred traffic converts 4.4x better than standard organic search, yet 27% of B2B sites accidentally block these high-value crawlers at the CDN layer. With 69% of AI crawlers unable to execute JavaScript and Gartner projecting a 25% drop in traditional search volume by 2026, brands must implement a selective robots.txt framework that blocks training bots like GPTBot while allowing search bots like OAI-SearchBot. Proper configuration ensures your content is cited in AI answers rather than being harvested without attribution or rendered invisible by client-side code. ### Mersel AI Platform and Navigation * [**Cite - Content engine:** Your dedicated website section that brings leads](/cite) * [**AI visibility analytics:** See which AI platforms visit your site and mention your brand](/platform/visibility-analytics) * [**Agent-optimized pages:** Show AI a version of your site built to get recommended](/platform/ai-optimized-pages) * **Live AI Traffic:** 3 AI visits today (GPTBotOptimized, ClaudeBotOptimized, PerplexityBotOptimized) * **System Status:** Chrome 122Original * **Links:** [Home](/) | [Blog](/blog) | [Pricing](/pricing) | [Login](https://app.mersel.ai) | Book an Audit Call | Book a Free Call | Platform | Language # How to Block AI Bots in robots.txt: GPTBot, ClaudeBot & More (2026) **Read Time:** 18 min read | **Author:** Mersel AI Team | **Date:** March 13, 2026 **The core strategic framework for AI bot management is to block AI training crawlers while allowing AI search crawlers.** Blanket blocking removes your brand from ChatGPT and Perplexity results entirely. Conversely, blanket allowing hands your proprietary content to model training datasets with no attribution, no backlinks, and no referral traffic in return. | AI Crawler Market Metric | Data Point | | :--- | :--- | | AI Bot Growth Rate | Doubled since August 2023 | | Cloudflare Global Protection | Roughly 20% of all websites | | Cloudflare Default Policy (2024) | Blocking AI crawlers by default on new domains | The number of active AI bots has doubled since August 2023, necessitating precise crawler management. Cloudflare, which protects roughly 20% of all websites globally, began blocking AI crawlers by default on new domains in 2024. Many technical SEO teams have perfectly configured `robots.txt` files that are being silently overridden at the CDN layer, resulting in accidental invisibility in the exact AI systems buyers use to build vendor shortlists. This guide provides the following resources for AI crawler optimization: * The exact `robots.txt` configuration to implement today. * A step-by-step process for auditing your CDN and rendering stack. * A clear framework for when to use `llms.txt` to further structure your content for AI extraction. ## Quick Answer: Which AI Bots to Block vs. Allow **Website owners should block training crawlers that harvest data without providing traffic while allowing search and citation crawlers that drive qualified users to the brand.** An AI crawler is an automated program operated by AI companies to harvest training data or fetch live content for user-facing answers. These two bot types have completely different impacts on visibility, requiring separate treatment within a site's `robots.txt` file. | Bot Category | User Agents | Impact and Recommended Action | | :--- | :--- | :--- | | **Training Crawlers** | GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, CCBot (Common Crawl), Meta-ExternalAgent, Bytespider, Applebot-Extended | **Block**: These bots harvest content for model training but do not send traffic back to the source website. | | **Search & Citation Crawlers** | OAI-SearchBot, ChatGPT-User (OpenAI), Claude-SearchBot, Claude-User (Anthropic), PerplexityBot, YouBot | **Allow**: These bots cite your brand and send qualified traffic through user-facing search and content fetches. | Access the full copy-paste `robots.txt` configuration in [Step 1 below](#step-1-configure-your-robotstxt-with-selective-access). For a comprehensive user agent reference table including official documentation links, refer to [the next section](#official-ai-bot-user-agent-reference-2026). ## Key Takeaways Training crawlers and search crawlers are distinct bots operated by the same company, and blocking one has zero effect on the functionality of the other. | Bot Name | Primary Function | Impact of Blocking | Source/Evidence | | :--- | :--- | :--- | :--- | | GPTBot | Trains OpenAI's models | No measurable impact on Google Search rankings | Publisher network analysis reviewed by Playwire | | OAI-SearchBot | Powers ChatGPT live search results | Removes site from ChatGPT search answers entirely | Direct system functionality | - **27% of B2B SaaS and ecommerce websites accidentally block major LLM crawlers** due to CDN-level rules, often without the owners' knowledge, according to research cited by ziptie.dev. These unintentional blocks prevent AI models from accessing critical business data and product information. - **69% of AI crawlers cannot execute JavaScript**, according to research by Vercel and MERJ. If a website relies heavily on client-side rendering, AI bots see a blank page regardless of robots.txt settings, effectively hiding the content from LLM discovery. - **AI-referred traffic converts 4.4x better than standard organic search**, according to data aggregated by Superlines. This significant performance gap makes visibility in AI search results a high-value pipeline source for modern digital marketing and sales strategies. - **llms.txt adoption sits at approximately 10% of domains**, according to Ahrefs. This emerging standard serves as a zero-risk, low-effort signal that effectively guides AI agents toward a website's highest-value content while improving overall machine readability. ## Why This Problem Keeps Getting Worse **Gartner projects that traditional search engine volume will drop 25% by 2026 as generative AI platforms absorb informational queries.** This shift is already visible in referral data: 60% of all Google searches end without a click, and organic click-through rates drop by up to 61% when a Google AI Overview appears for a query. Buyers who click from AI-generated answers are significantly more qualified because they have already consumed an AI-curated summary and evaluated alternatives. These users arrive at your site with high intent. However, you only capture this traffic if AI search bots can successfully read and cite your content in the first place. Most organizations fail to capture AI traffic for three reasons that have nothing to do with content quality: * **Reason 1: Treating all AI bots as one entity.** Brand managers often add a blanket `Disallow: /` for every user agent with "AI" or "Bot" in the name. This blocks `OAI-SearchBot` alongside `GPTBot`, removing the brand from ChatGPT's live search results entirely. * **Reason 2: CDN overrides of `robots.txt` instructions.** Cloudflare's AI blocking feature operates at the edge, returning a 403 Forbidden error to AI crawlers before the request reaches the origin server. A perfectly configured `robots.txt` is irrelevant when the firewall never lets the bot through. * **Reason 3: Site invisibility due to rendering limitations.** Major AI crawlers do not execute JavaScript, which creates a significant visibility gap compared to traditional search engines like Googlebot. | Feature | Googlebot | Major AI Crawlers | | :--- | :--- | :--- | | Rendering Engine | Full Chromium engine | No JavaScript execution | | SPA Compatibility | Executes React/Vue | Delivers blank `` | React or Vue single-page applications deliver a blank `` to AI bots, meaning your content simply does not exist for them. To understand the full scope of how AI bots discover and read web pages, see our guide on [what an AI bot crawler actually is and how it works](/blog/what-is-an-ai-bot-crawler). ## The Core Framework: Training Crawlers vs. Search Crawlers Major AI companies operate at least two distinct crawlers with completely separate functions, and confusing these bots is the root cause of most AI visibility failures. Training crawlers absorb content directly into model weights without providing attribution. In contrast, search crawlers retrieve live content specifically to cite it in user-facing answers. Blocking the wrong category of crawler often produces the opposite of the intended visibility effect for a website. | Feature | Training Crawlers | Search Crawlers | | :--- | :--- | :--- | | **Primary Function** | Absorb content into model weights | Retrieve live content for user answers | | **Attribution** | No attribution provided | Cites sources in user-facing answers | | **Impact of Blocking** | Protects intellectual property (IP) | Removes site from AI search results | OpenAI developer documentation explicitly distinguishes between its two primary bots: `OAI-SearchBot` and `GPTBot`. `OAI-SearchBot` surfaces websites within ChatGPT's search features, and sites that opt out will not appear in ChatGPT search answers. Conversely, `GPTBot` crawls content used for model training. OpenAI confirms that blocking `GPTBot` is an entirely independent action that does not impact a site's visibility in search results. Technical documentation from xseek.io highlights that these are independent systems, a key insight often missed by SEO teams. This separation allows webmasters to strategically manage their digital assets by blocking `GPTBot` to protect intellectual property while simultaneously allowing `OAI-SearchBot` to remain visible. This dual-track framework ensures that a brand maintains its presence in ChatGPT search results without contributing its data to underlying model training. ## Official AI Bot User Agent Reference (2026) Verified user agent strings for 2026 allow site owners to distinguish between AI training crawlers and search citation bots. Official documentation links serve as the authoritative source for these frequently updated strings, which dictate how AI models interact with web content. | AI company | Training crawler | Search / citation crawler | Recommended action | | --- | --- | --- | --- | | OpenAI | `GPTBot` ([docs](https://platform.openai.com/docs/bots)) | `OAI-SearchBot`, `ChatGPT-User` ([docs](https://platform.openai.com/docs/bots)) | Block GPTBot; allow OAI-SearchBot and ChatGPT-User | | Anthropic | `ClaudeBot` ([docs](https://support.anthropic.com/en/articles/8896518)) | `Claude-SearchBot`, `Claude-User` ([docs](https://support.anthropic.com/en/articles/8896518)) | Block ClaudeBot; allow Claude-SearchBot and Claude-User | | Google | `Google-Extended` ([docs](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)) | Uses Googlebot for AI Overviews | Block Google-Extended only — Googlebot still indexes for search | | Perplexity | None (no separate training crawler) | `PerplexityBot`, `Perplexity-User` ([docs](https://docs.perplexity.ai/guides/bots)) | Allow both | | Common Crawl | `CCBot` ([docs](https://commoncrawl.org/ccbot)) | N/A | Block — feeds many open-source LLM training sets | | Meta | `Meta-ExternalAgent`, `FacebookBot` | N/A | Block both | | ByteDance | `Bytespider` | N/A | Block | | Apple | `Applebot-Extended` | Uses Applebot for Spotlight / Siri search | Block Applebot-Extended only | | You.com | N/A | `YouBot` | Allow | **Anthropic requires specific user agent configurations to ensure effective blocking and search visibility.** Site owners must avoid deprecated strings like `Claude-Web` and `anthropic-ai` as they are no longer active. The current 2026 active strings include `ClaudeBot` for training, `Claude-SearchBot` for search indexing, and `Claude-User` for per-user fetches initiated by Claude.ai. ### Step 1: Configure Your `robots.txt` with Selective Access **Configuring a robots.txt file at the domain root provides the foundational layer for separating search bots from training crawlers.** This implementation ensures that intellectual property remains protected while maintaining visibility in Generative Engine Optimization (GEO) results. Place this file at `https://yourdomain.com/robots.txt` to explicitly define access permissions. ``` # -------------------------------------------------------- # 1. ALLOW AI Search & Retrieval (For GEO / Visibility) # -------------------------------------------------------- # OpenAI Search and User-Triggered Fetches User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # Anthropic Real-Time Fetches User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / # Perplexity AI Search User-agent: PerplexityBot Allow: / # You.com Search User-agent: YouBot Allow: / # -------------------------------------------------------- # 2. BLOCK AI Bulk Training Data Crawlers (IP Protection) # -------------------------------------------------------- # OpenAI Training User-agent: GPTBot Disallow: / # Anthropic Training User-agent: ClaudeBot Disallow: / # Google Generative AI Training (Does not impact Googlebot) User-agent: Google-Extended Disallow: / # Common Crawl (Used by many open-source LLMs) User-agent: CCBot Disallow: / # Meta/Facebook Training User-agent: Meta-ExternalAgent Disallow: / User-agent: FacebookBot Disallow: / # ByteDance/TikTok User-agent: Bytespider Disallow: / # -------------------------------------------------------- # 3. Standard Search Engines (Unchanged) # -------------------------------------------------------- User-agent: * Allow: / ``` **Robots.txt updates typically require approximately 24 hours for systems like OpenAI to process and adjust crawling behavior.** During this propagation period, existing crawler settings remain in effect. Administrators must verify that current active strings like `ClaudeBot` are targeted rather than inactive legacy strings such as `Claude-Web` or `anthropic-ai`. ### Step 2: Audit and Disable CDN-Level AI Blocking ## Technical Verification: CDN Configuration and Rendering Optimization **Verify that your CDN is not silently overriding your `robots.txt` configuration to prevent accidental AI invisibility.** This verification step is frequently overlooked by technical teams and accounts for the largest share of unintended site exclusion from AI models. If your website sits behind Cloudflare, Fastly, Shopify, or Wix, you must audit these settings to ensure your crawler instructions are being respected. **For Cloudflare users:** | Step | Action | | :--- | :--- | | 1 | Navigate to Security > Bots (or the "Control AI Crawlers" section) in your dashboard. | | 2 | Set "Block AI training bots" to allow crawlers, or configure WAF rules to explicitly allowlist OAI-SearchBot and PerplexityBot by user agent string. | | 3 | Verify that "Manage your robots.txt" inside Cloudflare is disabled so your origin server's file takes precedence. | **Approximately 27% of B2B SaaS and ecommerce websites are accidentally blocking major LLM crawlers at the CDN layer, according to research cited by ziptie.dev.** This misconfiguration often occurs even when the local `robots.txt` file is correctly configured. Auditing the CDN layer is essential for any site using managed platforms like Shopify or Wix to ensure AI search engines can access content. ### Step 3: Verify Bot Authentication Against IP Ranges **Authenticate legitimate AI search crawlers using published IP address ranges to prevent malicious scrapers from spoofing user agent strings.** Because `robots.txt` alone does not provide a complete defense against unauthorized access, you should use official JSON feeds within your WAF or bot management platform to validate requests. Use these feeds to accept real crawlers and reject spoofed requests claiming to be `OAI-SearchBot` from unauthorized IP ranges: * **OpenAI training crawler:** openai.com/gptbot.json * **OpenAI search crawler:** openai.com/searchbot.json ### Step 4: Fix the JavaScript Rendering Problem **Research by Vercel and MERJ reveals that 69% of AI crawlers cannot execute JavaScript, making client-side rendered content invisible to most models.** If your site is built using React, Vue, or Angular without server-side support, AI crawlers typically see a blank page. This rendering gap means your content remains unindexed regardless of your `robots.txt` permissions. **The fix for JavaScript invisibility involves three core components:** 1. **Server-side rendering (SSR):** Utilize frameworks such as Next.js or Nuxt that deliver fully rendered HTML in the initial response, allowing AI crawlers to parse content as simple HTTP clients. 2. **Semantic HTML structure:** Implement `
`, `
`, `
`, and `