---
title: "AI Crawler Explained for Local Business Owners"
description: "An AI crawler is a bot that visits your site to train a model, build a search index, or answer a user's question. See the three jobs and how you control them."
source: "https://www.lighthouselocal.ai/glossary/ai-crawler"
---

[Home](https://www.lighthouselocal.ai/) / [Glossary](https://www.lighthouselocal.ai/glossary) / AI crawler

# AI crawler

[Crawlers and access](https://www.lighthouselocal.ai/glossary#crawlers-and-access) · Also called AI bot

An AI crawler is an automated program an AI company sends to visit web pages, either to collect content for training its models, to build a search index, or to fetch a page when a user asks a question.

## Why it matters

AI assistants can only describe your business well if their systems can read your website. The programs that do the reading are AI crawlers, and your site decides which of them get in. The catch is that they don't all do the same job, so one blunt rule can switch off more than you meant to.

Knowing the three jobs helps you make that choice on purpose instead of by accident.

## Example

A family-run HVAC company asks its web developer to "block the AI bots" after reading a scary headline. The developer adds one rule for every AI crawler on a list. A few weeks later the owner asks an assistant for heating repair in her town and her company never comes up, because the crawlers that build search indexes were blocked along with the ones that collect training data. She asks the developer to rewrite the rules, blocking the training crawler only and leaving the search crawler and the question-triggered fetcher alone.

## The three jobs an AI crawler can do

An AI crawler is an automated visitor that identifies itself with a name called a user-agent. The companies that run them describe up to three jobs, each with its own bot names, and not every company runs all three.

The first is collecting content that may be used to train models. OpenAI says its GPTBot crawls content that may be used in training its generative AI foundation models, and Anthropic says ClaudeBot collects web content that could potentially contribute to training.

The second is building a search index. OpenAI says OAI-SearchBot is used to surface websites in search results in ChatGPT's search features, Anthropic says Claude-SearchBot navigates the web to improve search result quality, and Perplexity says PerplexityBot is designed to surface and link websites in search results.

The third is fetching a page because a person asked. OpenAI describes ChatGPT-User, Anthropic describes Claude-User, and Perplexity describes Perplexity-User as visiting a page when a user asks a question.

## How you control them

The main control is a robots.txt file, where you name a crawler's user-agent and say what it may visit. OpenAI says each of its settings is independent, so you can allow its search bot while disallowing GPTBot. Anthropic says its bots honor robots.txt directives and that blocking IP addresses may not work, because it stops them reading your robots.txt file.

The companies state their user-triggered fetchers' position separately. OpenAI says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. Perplexity says that, since a user requested the fetch, Perplexity-User generally ignores robots.txt rules. Anthropic says its bots, which include Claude-User, honor robots.txt directives.

Each company documents its own bot names, and they change, so check the operator's page before you write a rule.

## Sources

- [OpenAI Platform Docs: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots)
- [Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity Docs: Perplexity Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)

Related terms

- [robots.txt](https://www.lighthouselocal.ai/glossary/robots-txt)
- [Blocking AI crawlers](https://www.lighthouselocal.ai/glossary/blocking-ai-crawlers)
- [GPTBot](https://www.lighthouselocal.ai/glossary/gptbot)
- [ClaudeBot](https://www.lighthouselocal.ai/glossary/claudebot)

More in Crawlers and access

- [Google-Extended](https://www.lighthouselocal.ai/glossary/google-extended)
- [Indexing](https://www.lighthouselocal.ai/glossary/indexing)
- [llms.txt](https://www.lighthouselocal.ai/glossary/llms-txt)
- [PerplexityBot](https://www.lighthouselocal.ai/glossary/perplexitybot)
- [XML sitemap](https://www.lighthouselocal.ai/glossary/xml-sitemap)

## Ready to see how AI describes your business?

Set up in ten minutes with a 14-day free trial on every self-serve plan, or talk to a specialist first.

[Start Free Trial](https://app.lighthouselocal.ai/?utm_source=lighthouselocal&utm_medium=marketing-site&utm_campaign=glossary-ai-crawler&utm_content=closer) Book a Demo

## Sitemap

- [Markdown sitemap](https://www.lighthouselocal.ai/sitemap.md)
