Why it matters
AI assistants can only describe your business well if their systems can read your website. The programs that do the reading are AI crawlers, and your site decides which of them get in. The catch is that they don't all do the same job, so one blunt rule can switch off more than you meant to.
Knowing the three jobs helps you make that choice on purpose instead of by accident.
Example
A family-run HVAC company asks its web developer to "block the AI bots" after reading a scary headline. The developer adds one rule for every AI crawler on a list. A few weeks later the owner asks an assistant for heating repair in her town and her company never comes up, because the crawlers that build search indexes were blocked along with the ones that collect training data. She asks the developer to rewrite the rules, blocking the training crawler only and leaving the search crawler and the question-triggered fetcher alone.
The three jobs an AI crawler can do
An AI crawler is an automated visitor that identifies itself with a name called a user-agent. The companies that run them describe up to three jobs, each with its own bot names, and not every company runs all three.
The first is collecting content that may be used to train models. OpenAI says its GPTBot crawls content that may be used in training its generative AI foundation models, and Anthropic says ClaudeBot collects web content that could potentially contribute to training.
The second is building a search index. OpenAI says OAI-SearchBot is used to surface websites in search results in ChatGPT's search features, Anthropic says Claude-SearchBot navigates the web to improve search result quality, and Perplexity says PerplexityBot is designed to surface and link websites in search results.
The third is fetching a page because a person asked. OpenAI describes ChatGPT-User, Anthropic describes Claude-User, and Perplexity describes Perplexity-User as visiting a page when a user asks a question.
How you control them
The main control is a robots.txt file, where you name a crawler's user-agent and say what it may visit. OpenAI says each of its settings is independent, so you can allow its search bot while disallowing GPTBot. Anthropic says its bots honor robots.txt directives and that blocking IP addresses may not work, because it stops them reading your robots.txt file.
The companies state their user-triggered fetchers' position separately. OpenAI says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. Perplexity says that, since a user requested the fetch, Perplexity-User generally ignores robots.txt rules. Anthropic says its bots, which include Claude-User, honor robots.txt directives.
Each company documents its own bot names, and they change, so check the operator's page before you write a rule.