Why it matters
Some AI crawlers collect training data, and some fetch pages so an assistant can answer a customer's question. Blocking the first kind opts you out of training. Blocking the second kind can keep you out of the answers customers read.
Many owners never chose either one. A setting somebody switched on years ago may be doing it for you, so it's worth checking.
Example
A plumber's site shares a hosting plan with a security service that turns on a block for unknown bots. The plumber has never touched robots.txt, and the file looks fine. But the service refuses the bots that AI search tools send, so those tools can't read the new service pages. The plumber only finds out after asking an assistant about local plumbers and not being named. Allowing the right bots in the security settings, and keeping the training bots blocked if the plumber wants that, fixes it.
How a site ends up blocking AI crawlers
On purpose, the tool is robots.txt, a file that tells crawlers which URLs they can access on your site. Each operator publishes its own bot names. OpenAI says its settings are independent, so a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to say its content shouldn't be used for training. Anthropic and Perplexity publish named bots the same way.
By accident, the biggest risk is a broad rule. Google's robots.txt guide says an asterisk matches all crawlers except the AdsBot crawlers. A blanket disallow under it doesn't just shut out AI bots. It also shuts out Googlebot and Bingbot, and takes the site out of search.
The other route is infrastructure. Google's AI features page lists ensuring crawling is allowed in robots.txt and by any CDN or hosting infrastructure. Perplexity says that if you use a web application firewall you may need to explicitly allow its bots, and Anthropic says blocking IP addresses may not work correctly because it stops the bot from reading your robots.txt. Our guide to allowing our own audit crawler uses Lighthouse Local's crawler as the example and shows where firewall allow settings sit.
What it costs a local business
The cost depends on the bot. OpenAI says sites opted out of OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic says disabling Claude-SearchBot may reduce your visibility and accuracy in search results, and disabling Claude-User prevents retrieval of your content in response to a user's question. Perplexity recommends allowing PerplexityBot so your site appears in its search results.
Operators describe GPTBot and ClaudeBot as collecting content for training, so blocking one of those is an opt-out from training. Google-Extended is different because it also affects grounding in Gemini. And Googlebot, not Google-Extended, controls Google Search and its AI features, per Google.
A robots.txt block also isn't a guarantee against every fetch. OpenAI says robots.txt rules may not apply to ChatGPT-User because users initiate those actions, and Perplexity says Perplexity-User generally ignores robots.txt rules. Anthropic says its bots, Claude-User included, honor robots.txt. Google's guide adds that robots.txt instructions can't enforce crawler behavior.
Sources
- Google Search Central: Create and Submit a robots.txt File
- Google Search Central: Robots.txt Introduction and Guide
- Google Search Central: AI Features and Your Website
- OpenAI Platform Docs: Overview of OpenAI Crawlers
- Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity Docs: Perplexity Crawlers