Useful to test a file before uploading it, or one that sits behind a password.
The robots.txt is read by our Cloudflare function and not stored. What you paste and the URLs you test never leave your browser.
Result
—
Read a robots.txt or paste one to start.
The file
AI crawlers
What each one may do with the home page and with the tested URL, under this file.
| Crawler | Home (/) | Tested URL |
|---|---|---|
| Google-ExtendedTraining for Google’s AI models (Gemini). Blocking it does not affect Search. | ||
| Applebot-ExtendedTraining for Apple’s models. Blocking it does not affect Siri. | ||
| GPTBotOpenAI: model training. | ||
| ChatGPT-UserOpenAI: when a person asks ChatGPT to read a page. | ||
| OAI-SearchBotOpenAI: ChatGPT search results. Blocking it removes you from them. | ||
| ClaudeBotAnthropic: model training. | ||
| anthropic-aiAnthropic, legacy token. Worth listing alongside ClaudeBot. | ||
| Claude-WebAnthropic: when Claude reads a page on a person’s request. | ||
| PerplexityBotPerplexity: its search index. | ||
| Perplexity-UserPerplexity: reading on a person’s request. | ||
| CCBotCommon Crawl: the open archive many models are trained on. | ||
| BytespiderByteDance (TikTok). Known for ignoring robots.txt. | ||
| AmazonbotAmazon: Alexa and its models. | ||
| cohere-aiCohere: model training. | ||
| Meta-ExternalAgentMeta: training for its models (Llama). | ||
| DiffbotDiffbot: data extraction for third parties. |
Declared sitemaps
Findings
Next step
Shall we fix it for you?
Send us this result and we reply within one business day with what we would fix first, how much work it is and what you can leave as it is.
Opens the form with the diagnosis already written. You can edit it before sending.
How to use it
01
Read or paste the file
Type your domain and we read /robots.txt from our server, or paste the contents to test it before uploading.
02
Test a URL
Pick the crawler and the address. We tell you whether it may pass and which line decides, with the same longest-match rule Google uses.
03
Review the AI crawlers
The matrix shows what GPTBot, ClaudeBot and friends may do. Blocking them changes nothing on Google.
What robots.txt does, and what it does not
It tells crawlers what they may request. Nothing more. It does not deindex: a blocked URL that receives links can show up on Google with a half title and no description, because Google knows it exists but cannot read it. To remove something from the index you use the "noindex" robots meta tag or the X-Robots-Tag header, and for that the crawler has to be able to get in.
Nor is it access control: it is a courtesy request that serious crawlers honour and the rest do not. Whatever you genuinely do not want read has to sit behind a password.
Blocking AI crawlers
Each AI company uses a different crawler to train its models and, sometimes, another to answer when a person asks it to read a page. GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent and Bytespider collect text for training; ChatGPT-User and Claude-Web act on someone’s request. Google-Extended is not a crawler: it is a switch telling Googlebot not to use what it reads to train Gemini.
Blocking the training ones changes nothing in Google or Bing search results. What it does change is your presence in AI search engines: OAI-SearchBot feeds ChatGPT search and PerplexityBot feeds Perplexity. Decide separately what you want for each.
A broken robots.txt shows up late
The typical case: a "Disallow: /" that was set on staging and shipped to production with the deploy. Nobody notices, because the site works. What happens is that Google stops crawling, and rankings drop weeks later, when nobody remembers the deploy.
On the sites we build, the robots.txt is generated by the build according to the environment and verified before publishing: on staging it blocks everything and on production it blocks nothing Google needs. It is the difference between finding out in the build and finding out in Search Console.
Frequently asked questions
Does blocking a URL in robots.txt remove it from Google?
No. robots.txt controls crawling, not indexing. If other pages link to that URL, Google can still list it, without title or description, because it cannot read it. To remove it from the index you have to let it be crawled and add a "noindex" robots meta tag or the X-Robots-Tag header. Blocking it in robots.txt on top of noindex is counterproductive: Google never gets to see the noindex.
How do I block ChatGPT without losing visibility on Google?
With a dedicated group for AI crawlers: "User-agent: GPTBot", "User-agent: ClaudeBot", "User-agent: CCBot" and "Disallow: /" below. Google uses Googlebot for Search and Google-Extended only to train its models, so blocking Google-Extended does not remove you from results either. Note that OAI-SearchBot feeds ChatGPT search: blocking it does remove you from there.
What happens if I have no robots.txt?
Nothing bad: a 404 on /robots.txt means "everything allowed" and is perfectly normal. What does hurt is a 5xx error on that path, because Google reads it as "I do not know whether I may crawl" and stops for a while.
What does an empty "Disallow:" mean?
That nothing is forbidden: it is the same as having no rule. Many people write it thinking it blocks everything, and it is exactly the opposite. To block everything it is "Disallow: /".
Is it case sensitive?
Paths are: "Disallow: /Admin/" does not block "/admin/". Directive and crawler names are not: "user-agent: googlebot" works the same as "User-agent: Googlebot".
Which wildcards can I use?
Two, and only two: the asterisk, which matches any sequence of characters, and the dollar sign at the end, which requires the path to end there. "Disallow: /*.pdf$" blocks PDFs; without the dollar it would also block "/report.pdf?v=2". There are no regular expressions.
Does "Crawl-delay" work?
With Google, no: it has always ignored it, and crawl rate is adjusted from Search Console. Bing and Yandex do respect it. If your problem is a crawler overloading the server, the right place to throttle it is the server or the CDN, not robots.txt.
Is anything I test stored?
No. The robots.txt is read by our Cloudflare function when you type a domain, returned to your browser and not stored; it is cached for ten minutes so it is not requested twice in a row. What you paste by hand and the URLs you test never leave your browser.
Is Google crawling what it should, and nothing else?
A good robots.txt is one line. Knowing what Google really crawls, what you block by accident and what slips through to AI crawlers is work we do in every technical audit.
We reply within one business dayBogotá · UTC−5