Why it matters
Robots.txt is the first place a crawler looks to learn what you'd like it to leave alone. It's how you let a search crawler in while keeping a training crawler out, because each rule names a crawler and says which paths it may visit.
It's a request, not a lock. It isn't a way to hide a page, and it isn't security. A page you block can still show up if other sites link to it, and anything private needs a real protection like a password.
Example
A dental practice's website was built years ago, and nobody remembers what its robots.txt says. The office manager opens the practice's address followed by /robots.txt in a browser and finds a short list of lines. One of them tells every crawler to stay out of the whole site, left over from when the site was being built. She asks her developer to remove it, then adds a separate rule for one specific AI crawler she does want to limit.
What the file is and where it lives
Robots.txt follows a standard called the Robots Exclusion Protocol, published as RFC 9309. It says the rules must be in a file named "/robots.txt" (all lowercase) in the top-level path of the service. That means a site at one address has its file at the root of that address, not in a folder.
The file is read in groups. A group starts with a user-agent line naming a crawler and is followed by allow and disallow rules for paths. The standard says that if no rule matches for a crawler, the URL is allowed, and that the most specific match is used.
What it can and can't do
RFC 9309 says plainly that these rules aren't a form of access authorization, and that the protocol isn't a substitute for valid content security measures. Google's documentation adds that a robots.txt file isn't a mechanism for keeping a web page out of Google, and that a blocked page's URL can still appear in search results. To keep a page out of Google, Google says to use noindex or password protection instead.
For AI crawlers, the same file is where you name each bot. Anthropic's help article shows blocking ClaudeBot from a whole site with a user-agent line followed by a disallow rule of a single slash, and asks you to do it for every subdomain you want to opt out. Each AI company documents its own crawler names, so check its page before you write a rule.