Is your website blocking AI search engines?

Last updated

Updated November 2026.

Many websites block AI search engines without knowing it – through a line in robots.txt, a firewall rule or a Cloudflare setting. If you want to be recommended in ChatGPT, Perplexity and Claude, their search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) need access. You can say no to your content being used for training (GPTBot, ClaudeBot, Google-Extended) and still appear in AI answers.

Training and search are two different things

AI companies use different crawlers for different purposes:

  • Training crawlers collect content that may be used to train future AI models. Blocking them doesn’t affect whether you appear in search answers.
  • Search crawlers index pages so they can be shown and linked in the assistant’s answers. Block them and you disappear from those answers.
  • User fetchers retrieve a page when a user asks the assistant to. They don’t always follow robots.txt, because a person requested the visit.

So if you block everything with one rule, you lose visibility too.

The most important crawlers

The names below were checked against the official documentation in November 2026. They change now and then, so check the sources before you change anything.

OpenAI (documentation)

  • GPTBot – training. Blocking it means the content should not be used to train the models.
  • OAI-SearchBot – search in ChatGPT. Block it and the site won’t be shown in ChatGPT’s search answers.
  • ChatGPT-User – visits when a user asks for it. robots.txt may not apply.

Anthropic (documentation)

  • ClaudeBot – training.
  • Claude-SearchBot – search. Blocking it reduces visibility in Claude’s search answers.
  • Claude-User – visits on behalf of a user.

Perplexity (documentation)

  • PerplexityBot – shows and links pages in Perplexity’s search results. According to Perplexity, not used for training.
  • Perplexity-User – visits when a user asks. Generally ignores robots.txt.

Google (documentation)

  • Googlebot – Google Search, and therefore also AI Overviews and AI Mode, which build on the search index. Never block it if you want to be found on Google.
  • Google-Extended – not a separate crawler but a robots.txt token that controls whether your content may be used to train Gemini and to ground Gemini answers. Google says it does not affect your inclusion or ranking in Google Search.

Microsoft

  • Bingbot – Bing’s search index, which Microsoft Copilot builds on. Block Bingbot and you’ll be hard to find in Copilot too.

Example robots.txt

If you want to allow search but say no to training, it could look like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

The search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) fall under User-agent: * and so have access. Consider whether you want to block training at all – it’s a choice, not a requirement.

Cloudflare and other firewalls

robots.txt is only half of it. If you use Cloudflare, check Security → Settings → Bot traffic → AI Crawler and Agent access policies. There you set a rule for each of three types: Search, Agent and Training. According to Cloudflare’s documentation the default is “Allow” for all three, but the setting may have been changed during setup – by you, your agency or a previous developer.

If Search or Agent is set to “Block”, ChatGPT and Perplexity can’t read your site, whatever robots.txt says. The same goes for WordPress security plugins and other “bot protection” at your web host.

How to check

  1. Open your robots.txt at yourdomain.dk/robots.txt. Look for Disallow: / under User-agent: * or under the names above.
  2. Check Cloudflare or your web host for AI bot blocking, “bot fight mode” and similar.
  3. Check your server logs for visits from OAI-SearchBot, PerplexityBot and Claude-SearchBot. If they never come by, or get error codes (403), something is blocked.
  4. Check Google Search Console with URL Inspection that Googlebot can fetch the page.
  5. Ask the AI. Ask ChatGPT or Perplexity to summarise a specific page on your domain. If they can’t, that’s a sign.

It’s not only about access

Open access is a prerequisite, not a guarantee. Once crawlers can read the site, the content also has to say clearly what you do and where. Read How to get your business recommended by ChatGPT.

BusinessVeno, which we built, shows whether AI assistants mention your business and points to technical problems keeping them out. Try the free AI check, no account needed, or read about our work with GEO and AI visibility and hosting and operations.