---
title: "ClaudeBot Explained: Anthropic's Crawler for Your Website"
description: "ClaudeBot is Anthropic's crawler for model training. See what it does, how it differs from Claude-User and Claude-SearchBot, and how robots.txt controls it."
source: "https://www.lighthouselocal.ai/glossary/claudebot"
---

[Home](https://www.lighthouselocal.ai/) / [Glossary](https://www.lighthouselocal.ai/glossary) / ClaudeBot

# ClaudeBot

[Crawlers and access](https://www.lighthouselocal.ai/glossary#crawlers-and-access)

ClaudeBot is Anthropic's web crawler that collects web content which could potentially contribute to training its Claude AI models, and you control it with a rule in your robots.txt file.

## Why it matters

ClaudeBot is the Anthropic crawler tied to model training. Restricting it signals that your site's future materials should be left out of Anthropic's training datasets. It's separate from the bots Anthropic uses to search the web and to answer a user's question.

If you want Claude to be able to find and describe your business, you'll want to understand which of those bots you're limiting before you add a rule.

## Example

The owner of a small accounting firm reads that Anthropic runs three different bots. She doesn't want her articles used for training, but she does want Claude to be able to look up her firm when a client asks. She asks her web developer for a rule naming ClaudeBot only, and to leave Claude-SearchBot and Claude-User unblocked.

## What Anthropic says ClaudeBot does

Anthropic's help article says ClaudeBot helps enhance the utility and safety of its generative AI models by collecting web content that could potentially contribute to their training. When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from Anthropic's AI model training datasets.

To block it from your whole site, Anthropic shows a robots.txt file in your top-level directory with a user-agent line naming ClaudeBot and a disallow rule of a single slash. It asks you to do this for every subdomain you want to opt out. Anthropic says its bots honor robots.txt, and that blocking IP addresses may not work, because it stops them reading your robots.txt file.

## Anthropic's other agents

Anthropic documents three bots. Besides ClaudeBot, Claude-User supports Claude users. When people ask Claude questions, it may access websites using a Claude-User agent. Anthropic says disabling it prevents its system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.

Claude-SearchBot navigates the web to improve search result quality for users. Anthropic says disabling it prevents its system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.

Anthropic also says it supports the non-standard Crawl-delay extension to robots.txt, and that it has begun honoring a general no-AI-training signal, which it's rolling out across its services over time. If your robots.txt names ClaudeBot, Anthropic says it follows that rule even if the general setting differs. Anthropic updates its article, so check it before you edit your file.

## Sources

- [Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

Related terms

- [AI crawler](https://www.lighthouselocal.ai/glossary/ai-crawler)
- [robots.txt](https://www.lighthouselocal.ai/glossary/robots-txt)
- [Blocking AI crawlers](https://www.lighthouselocal.ai/glossary/blocking-ai-crawlers)
- [GPTBot](https://www.lighthouselocal.ai/glossary/gptbot)

More in Crawlers and access

- [Google-Extended](https://www.lighthouselocal.ai/glossary/google-extended)
- [Indexing](https://www.lighthouselocal.ai/glossary/indexing)
- [llms.txt](https://www.lighthouselocal.ai/glossary/llms-txt)
- [PerplexityBot](https://www.lighthouselocal.ai/glossary/perplexitybot)
- [XML sitemap](https://www.lighthouselocal.ai/glossary/xml-sitemap)

## Ready to see how AI describes your business?

Set up in ten minutes with a 14-day free trial on every self-serve plan, or talk to a specialist first.

[Start Free Trial](https://app.lighthouselocal.ai/?utm_source=lighthouselocal&utm_medium=marketing-site&utm_campaign=glossary-claudebot&utm_content=closer) Book a Demo

## Sitemap

- [Markdown sitemap](https://www.lighthouselocal.ai/sitemap.md)
