---
title: "GPTBot Explained: OpenAI's Crawler for Your Website"
description: "GPTBot is OpenAI's crawler for model training. See what it does, how it differs from OAI-SearchBot and ChatGPT-User, and how robots.txt controls it."
source: "https://www.lighthouselocal.ai/glossary/gptbot"
---

[Home](https://www.lighthouselocal.ai/) / [Glossary](https://www.lighthouselocal.ai/glossary) / GPTBot

# GPTBot

[Crawlers and access](https://www.lighthouselocal.ai/glossary#crawlers-and-access)

GPTBot is OpenAI's web crawler that collects content which may be used to train its generative AI foundation models, and you control it with a rule in your robots.txt file.

## Why it matters

GPTBot is the OpenAI crawler that OpenAI ties to model training. Blocking it tells OpenAI your content shouldn't be used for that purpose. It doesn't decide whether ChatGPT can show your business in search answers, because OpenAI uses a different crawler for that.

That split is the part owners miss. If you block the wrong bot, or every bot, you can opt out of more than you intended.

## Example

A landscaping company owner doesn't want her site used to train AI models, but she does want ChatGPT to be able to find her when someone asks for a lawn care company nearby. Her developer adds a rule for GPTBot only and leaves OAI-SearchBot alone. A month later she asks ChatGPT about lawn care in her town and checks whether her business can still appear.

## What OpenAI says GPTBot does

OpenAI's crawler documentation says GPTBot is used to make its generative AI foundation models more useful and safe, and that it crawls content that may be used in training those models. It says disallowing GPTBot indicates a site's content shouldn't be used in training generative AI foundation models. The user-agent token is GPTBot, and OpenAI publishes its IP addresses at a JSON file linked from the page.

To limit it, you add a robots.txt rule that names GPTBot. OpenAI says its settings are independent of each other, so a rule for one bot doesn't change what another may do.

## OpenAI's other agents

OpenAI documents more than one agent. OAI-SearchBot is used to surface websites in search results in ChatGPT's search features. OpenAI says sites opted out of OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links, and it recommends allowing OAI-SearchBot if you want to appear in search results.

ChatGPT-User is used for certain user actions in ChatGPT and Custom GPTs. When users ask a question, it may visit a web page. OpenAI says it isn't used for crawling the web in an automatic fashion, and because a user starts these actions, robots.txt rules may not apply. OpenAI says to use OAI-SearchBot in robots.txt for managing search opt outs and automatic crawl.

OpenAI says that for search results it can take about 24 hours from a robots.txt update for its systems to adjust. Bot names and behavior change, so check OpenAI's page before you edit your file.

## Sources

- [OpenAI Platform Docs: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots)

Related terms

- [AI crawler](https://www.lighthouselocal.ai/glossary/ai-crawler)
- [robots.txt](https://www.lighthouselocal.ai/glossary/robots-txt)
- [ChatGPT](https://www.lighthouselocal.ai/glossary/chatgpt)
- [Blocking AI crawlers](https://www.lighthouselocal.ai/glossary/blocking-ai-crawlers)

More in Crawlers and access

- [ClaudeBot](https://www.lighthouselocal.ai/glossary/claudebot)
- [Google-Extended](https://www.lighthouselocal.ai/glossary/google-extended)
- [Indexing](https://www.lighthouselocal.ai/glossary/indexing)
- [llms.txt](https://www.lighthouselocal.ai/glossary/llms-txt)
- [PerplexityBot](https://www.lighthouselocal.ai/glossary/perplexitybot)
- [XML sitemap](https://www.lighthouselocal.ai/glossary/xml-sitemap)

## Ready to see how AI describes your business?

Set up in ten minutes with a 14-day free trial on every self-serve plan, or talk to a specialist first.

[Start Free Trial](https://app.lighthouselocal.ai/?utm_source=lighthouselocal&utm_medium=marketing-site&utm_campaign=glossary-gptbot&utm_content=closer) Book a Demo

## Sitemap

- [Markdown sitemap](https://www.lighthouselocal.ai/sitemap.md)
