PerplexityBot

Crawlers and access

PerplexityBot is Perplexity's web crawler, designed to surface and link websites in Perplexity's search results, and you control it with a rule in your robots.txt file.

Why it matters

Unlike the crawlers built to collect training data, PerplexityBot is about search. Perplexity says it's designed to surface and link websites in its search results and isn't used to crawl content for AI foundation models. If you block it, you're asking Perplexity not to surface your site there.

So the question for a local business isn't how to keep it out. It's whether you want to show up when someone asks Perplexity for a business like yours.

Example

A pet groomer's developer blocked every unfamiliar bot on the site last year to cut down on junk traffic. The groomer asks Perplexity for dog groomers in her town and never sees her own shop. She asks the developer to check the robots.txt file, finds PerplexityBot was caught in the block, and has the rule removed so her site can be surfaced again.

What Perplexity says PerplexityBot does

Perplexity's crawler page says PerplexityBot is designed to surface and link websites in search results on Perplexity, and that it isn't used to crawl content for AI foundation models. The user-agent token is PerplexityBot, and the page gives an example user-agent string and a link to the published IP addresses.

Perplexity recommends allowing PerplexityBot in your robots.txt file and permitting requests from its published IP ranges if you want your site to appear in search results. It says each setting works independently and that it may take up to 24 hours for its systems to reflect changes.

Perplexity-User

Perplexity documents a second agent, Perplexity-User, which supports user actions within Perplexity. When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response. Perplexity says it isn't used for web crawling or to collect content for training AI foundation models.

Perplexity also says that since a user requested the fetch, this fetcher generally ignores robots.txt rules. The page adds firewall setup steps for allowing both bots if you use a web application firewall. Bot details change, so check Perplexity's page before you edit your file.

Sources

Ready to see how AI describes your business?

Set up in ten minutes with a 14-day free trial on every self-serve plan, or talk to a specialist first.

Start Free Trial