Blog

Is your robots.txt hiding you from AI search?

Lots of websites block AI crawlers without the owner knowing. Some of those crawlers train models. Others fetch your pages so an AI tool can recommend you. You can block the first kind and still allow the second.

Check yours in a minute

Open your website's address with /robots.txt on the end, for example yourbusiness.com/robots.txt. It's a short text file of rules for crawlers. Look for lines like these:

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /

Disallow: / under a name blocks that crawler from the whole site. Under User-agent: * it blocks every crawler that follows the rules, search engines included. If you see that and you didn't mean it, it's worth talking to whoever runs your site.

Robots.txt isn't the only place this happens. Some hosts and security services let you block AI crawlers with a single switch, and some now do it by default. Cloudflare, for example, started blocking known AI crawlers by default on new sites in 2025. If you use one, check its bot settings too.

Not all AI crawlers do the same job

The big AI companies now use separate crawlers for separate jobs. Roughly:

  • Training crawlers collect pages to train future models. GPTBot (OpenAI), ClaudeBot (Anthropic) and Meta-ExternalAgent (Meta) are examples.
  • Search crawlers index pages so the AI tool can find and cite them when someone asks a question. OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude) and PerplexityBot (Perplexity) are examples.
  • User fetchers visit a page because someone asked the tool to look at it, for example ChatGPT-User, Claude-User and Perplexity-User.

Google works differently. Googlebot crawls for Google Search, and a separate rule called Google-Extended controls whether Google may use your pages for its Gemini models. Blocking Google-Extended doesn't remove you from Google Search.

The names and what each one covers do change, so check each company's own crawler documentation before you rely on this list. This is how they describe them as of November 2026.

Block training, allow search

If you'd rather your pages didn't train AI models but you still want AI search tools to find you, a robots.txt like this does that for the crawlers above:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Whether to allow training is your call, and reasonable people choose differently. The point is to choose on purpose. Many sites block everything AI-related because a setting looked sensible, and that includes the crawlers that help customers find them.

What allowing crawlers won't do

Letting AI search crawlers in doesn't mean you'll be recommended. It means the tools can read your site when they look. Whether they name you still depends on what your pages say and what the rest of the web says about you. We wrote about that here.

Also, robots.txt is a request, not a lock. Reputable crawlers follow it. Anything private shouldn't be on a public page at all.

Find your dot.