---
title: "robots.txt Explained for Local Business Websites"
description: "robots.txt is a plain-text file that tells crawlers, including AI crawlers, which parts of your site to visit. See what it can do and what it can't."
source: "https://www.lighthouselocal.ai/glossary/robots-txt"
---

[Home](https://www.lighthouselocal.ai/) / [Glossary](https://www.lighthouselocal.ai/glossary) / robots.txt

# robots.txt

[Crawlers and access](https://www.lighthouselocal.ai/glossary#crawlers-and-access)

A robots.txt file is a plain-text file at the top level of your website that asks crawlers, including AI crawlers, which parts of your site they may visit.

## Why it matters

Robots.txt is the first place a crawler looks to learn what you'd like it to leave alone. It's how you let a search crawler in while keeping a training crawler out, because each rule names a crawler and says which paths it may visit.

It's a request, not a lock. It isn't a way to hide a page, and it isn't security. A page you block can still show up if other sites link to it, and anything private needs a real protection like a password.

## Example

A dental practice's website was built years ago, and nobody remembers what its robots.txt says. The office manager opens the practice's address followed by /robots.txt in a browser and finds a short list of lines. One of them tells every crawler to stay out of the whole site, left over from when the site was being built. She asks her developer to remove it, then adds a separate rule for one specific AI crawler she does want to limit.

## What the file is and where it lives

Robots.txt follows a standard called the Robots Exclusion Protocol, published as RFC 9309. It says the rules must be in a file named "/robots.txt" (all lowercase) in the top-level path of the service. That means a site at one address has its file at the root of that address, not in a folder.

The file is read in groups. A group starts with a user-agent line naming a crawler and is followed by allow and disallow rules for paths. The standard says that if no rule matches for a crawler, the URL is allowed, and that the most specific match is used.

## What it can and can't do

RFC 9309 says plainly that these rules aren't a form of access authorization, and that the protocol isn't a substitute for valid content security measures. Google's documentation adds that a robots.txt file isn't a mechanism for keeping a web page out of Google, and that a blocked page's URL can still appear in search results. To keep a page out of Google, Google says to use noindex or password protection instead.

For AI crawlers, the same file is where you name each bot. Anthropic's help article shows blocking ClaudeBot from a whole site with a user-agent line followed by a disallow rule of a single slash, and asks you to do it for every subdomain you want to opt out. Each AI company documents its own crawler names, so check its page before you write a rule.

## Sources

- [RFC Editor: Robots Exclusion Protocol (RFC 9309)](https://www.rfc-editor.org/rfc/rfc9309.html)
- [Google Search Central: Robots.txt Introduction and Guide](https://developers.google.com/search/docs/crawling-indexing/robots/intro)
- [Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

Related terms

- [AI crawler](https://www.lighthouselocal.ai/glossary/ai-crawler)
- [Blocking AI crawlers](https://www.lighthouselocal.ai/glossary/blocking-ai-crawlers)
- [XML sitemap](https://www.lighthouselocal.ai/glossary/xml-sitemap)
- [llms.txt](https://www.lighthouselocal.ai/glossary/llms-txt)

More in Crawlers and access

- [ClaudeBot](https://www.lighthouselocal.ai/glossary/claudebot)
- [Google-Extended](https://www.lighthouselocal.ai/glossary/google-extended)
- [GPTBot](https://www.lighthouselocal.ai/glossary/gptbot)
- [Indexing](https://www.lighthouselocal.ai/glossary/indexing)
- [PerplexityBot](https://www.lighthouselocal.ai/glossary/perplexitybot)

## Ready to see how AI describes your business?

Set up in ten minutes with a 14-day free trial on every self-serve plan, or talk to a specialist first.

[Start Free Trial](https://app.lighthouselocal.ai/?utm_source=lighthouselocal&utm_medium=marketing-site&utm_campaign=glossary-robots-txt&utm_content=closer) Book a Demo

## Sitemap

- [Markdown sitemap](https://www.lighthouselocal.ai/sitemap.md)
