What robots.txt does, and how to decide what to do about AI crawlers
How a robots.txt file is written, why blocking a URL does not remove it from Google, which rule wins when two disagree, and what your options are with GPTBot, ClaudeBot and the rest.
By Javier Tenjo, Full stack developer
On this page
robots.txt is a plain text file that sits at the root of your domain (yourdomain.com/robots.txt) and tells each crawler which parts of the site it may visit. It has worked as a gentlemen’s agreement since the nineties, and since 2022 it finally has a formal specification, RFC 9309.
For a long time it was a file you wrote once and forgot about. That changed when the crawlers of artificial intelligence companies arrived: there is now a business decision to make, and this is where it gets made.
How the file is written
The file is organised into groups. Each group starts with one or more User-agent lines, naming the robot it applies to, followed by Disallow rules (stay out of here) and Allow rules (this is fine):
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help/
User-agent: GPTBot
Disallow: /
Sitemap: https://yourdomain.com/sitemap.xmlA few reading rules worth having straight:
- Paths are case-sensitive.
Disallow: /Blog/does not block/blog/. - A rule is a prefix.
Disallow: /searchblocks/search,/search/and also/search-our-stores. If you only meant the folder, write/search/. - There are two wildcards.
*stands for any sequence of characters and$marks the end of the URL:Disallow: /*.pdf$blocks every PDF. - An empty
Disallow:blocks nothing. Closing the whole site isDisallow: /, a one-character difference that has kept more than one site out of Google. - The
Sitemapline belongs to no group and can go anywhere. It is the simplest way to say where your sitemap lives.
Google reads at most the first 500 KB of the file and discards the rest. A robots.txt anywhere near that size is almost always generating more rules than it needs.
Which rule wins when two disagree?
This is where most mistakes happen, because intuition says order matters, and it does not.
First, the group is chosen. A robot looks for the group whose User-agent matches its name most specifically. If there is a Googlebot group, Googlebot obeys only that one and ignores the * group entirely. Groups do not add up: if you blocked /admin/ under * and then created a Googlebot group without repeating that rule, Googlebot may go into /admin/.
Then, within the group, the longest rule wins. With Disallow: /search and Allow: /search/help/, the URL /search/help/contact is allowed because the Allow rule is more specific. If two rules have the same length and contradict each other, Google applies the least restrictive one, which is the Allow.
Whatever is not forbidden is allowed. If no rule matches the URL, the robot may crawl it. And if the file does not exist and the server returns 404, everything is allowed for everyone.
Disallow is not noindex
This is the most expensive misunderstanding in the whole file. Disallow stops a page from being crawled, not from being indexed. If a blocked URL receives links, Google can show it in results without ever having read it, with a title taken from the link text and no description.
Keeping a page out of Google takes the opposite: Google must be able to crawl it and find a noindex instruction, either in a <meta name="robots" content="noindex"> tag or in the X-Robots-Tag HTTP header. Block it in robots.txt and Google never gets to see that noindex.
For years there was a habit of writing Noindex: inside robots.txt itself. Google stopped supporting it in 2019, so that line does nothing today.
The practical rule:
| What you want | What you use |
|---|---|
| Crawlers not to waste time on an area (filters, internal search) | Disallow in robots.txt |
| A page not to appear in results | noindex on the page, without blocking it |
| Something not to be reachable by anyone | A password or a server-side restriction |
That last point bears repeating: robots.txt is public. Listing /secret-internal-panel/ in it announces its existence to anyone who opens the file.
What not to block
One mistake keeps turning up on older sites: blocking the CSS and JavaScript folders. Google renders pages the way a browser does, and if it cannot download the styles and scripts, it evaluates a broken version of your site. The same goes for images you want to appear in Google Images.
It is also worth reviewing the file after every migration or go-live. A Disallow: / that made sense on the staging environment and was copied as-is to the live site is a classic that can take weeks to spot.
Who the AI crawlers are
Artificial intelligence crawlers identify themselves with their own User-agent, and most companies publish the list. The key point is that they do not all do the same thing. There are three distinct jobs:
- Model training. They collect content to train future versions. This is where
GPTBot(OpenAI),ClaudeBot(Anthropic),CCBot(Common Crawl, an open archive many models use),Meta-ExternalAgentandApplebot-Extendedbelong. - Search inside the assistant. They index pages to cite them when someone asks a question.
OAI-SearchBot, for instance, feeds ChatGPT’s search answers, andPerplexityBotfeeds Perplexity’s index. - Reading on request. They visit a URL because a person asked the assistant to, right then, as
ChatGPT-UserorPerplexity-Userdo.
Google-Extended is a special case. It is not a robot that crawls on its own: it is a token Google checks to know whether it may use your content to train its Gemini models. Blocking it does not affect your presence in Google Search, which is still handled by Googlebot.
Names change and new ones appear fairly often, so before writing rules it pays to check each company’s documentation.
Choosing a policy
There is no answer that is right for everyone, but there are three reasonable positions:
- Allow everything. It makes sense if your business depends on being found and cited: services, retail, educational content. It is what happens if you do nothing.
- Block training, allow search. You block
GPTBot,ClaudeBot,CCBot,Google-Extendedand the like, and let search and on-request robots through. Your content does not feed future models, but you can still be cited when someone asks. - Block all AI. It suits publishers, media and businesses whose value lies precisely in their content. The cost is disappearing from those assistants’ answers.
Bear one limitation in mind: robots.txt is a request, not a barrier. The big companies’ crawlers respect it, but some robots ignore it. If you need to enforce your decision, the tool for that is the server or the content delivery network (a CDN), which can reject requests by User-agent or by behaviour.
Checking yours
Reading a robots.txt by eye works until there are three groups and a dozen rules. The robots.txt tester reads the file of any domain, or one you paste before uploading it, and tells you whether a given URL is allowed or blocked for each robot, with the exact line that decides it. It includes a table of the main AI crawlers so you can see at a glance what each one may do, and it flags the usual mistakes: a useless Noindex, blocked CSS, or a * group that shuts the whole site.
If you want to understand what a search engine does after crawling, it is covered in how a search engine works.
robots.txt is a small file with large consequences. Review it every time you change platforms, make a deliberate choice about AI crawlers, and remember that to take something out of Google the tool is noindex, not Disallow.