Test your robots.txt against every crawler that matters.

What robots.txt does, and what it doesn’t

robots.txt is a plain-text file at the root of a host, such as https://www.example.com/robots.txt, that tells crawlers which URLs they may fetch. It applies only to the protocol, host and port it is served from, so the file on www doesn’t cover the bare domain, a subdomain or the http version.

It controls crawling, not indexing. A blocked URL can still be indexed if other pages link to it; it just appears without a description, because Google can’t read the page. To keep a page out of search results, let Google crawl it and add noindex, or put it behind a password.

It is also a request, not a lock. Well-behaved crawlers such as Googlebot follow it, but RFC 9309, the standard behind the file, says its rules are not a form of access authorisation. Listing a private path there only advertises it.

How Google reads the rules

The file is made of groups, each starting with one or more User-agent lines followed by Allow and Disallow rules. A crawler follows only the group that names it most specifically. Groups for the same crawler are merged, and the * group applies only to crawlers with no group of their own: add a Googlebot group and Googlebot ignores everything under User-agent: *.

Within that group, the most specific rule wins, judged by the length of its path, and if an Allow and a Disallow rule are equally specific, Google uses the less restrictive Allow. * matches any sequence of characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks /guide.pdf but not /guide.pdf?v=2. Paths are case-sensitive.

Google supports only user-agent, allow, disallow and sitemap, so crawl-delay and noindex lines do nothing, and it ignores anything beyond the first 500 KiB. It generally caches the file for up to 24 hours. A 4xx response other than 429 is treated as if there were no file. A 5xx response or an unreachable file stops crawling for 12 hours; after that, Google works from its last good copy for up to 30 days.

Common mistakes and how to fix them

In audits we see the same problems again and again:

• A Disallow: / left over from a staging site. Remove it at launch, and protect staging with a password instead.
• Blocked CSS or JavaScript. If Google needs those files to understand a page, blocking them stops it analysing the page properly.
• Blocking pages to get them out of results. Lift the block, add noindex and let Google recrawl them.
• A specific group that drops the general rules. Copy into the Googlebot group anything from * that Googlebot should still follow.
• Rules written for the wrong case or host. Disallow: /Checkout doesn’t block /checkout, and every subdomain needs its own file.
• A robots.txt that returns a server error, often because of a firewall or CDN rule, which can pause crawling of the whole site.

AI crawlers: training, search and user requests

AI companies run several bots with different jobs. Training crawlers, such as OpenAI’s GPTBot and Anthropic’s ClaudeBot, collect content that may be used to train models. Search crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, find the pages that AI search tools show and link to. User-triggered fetchers, such as ChatGPT-User, Claude-User and Perplexity-User, fetch a page when someone asks for it.

Some names are tokens, not crawlers. Google-Extended has no crawler of its own: disallowing it opts your content out of training future Gemini models and of grounding in Gemini apps and Vertex AI, and Google says it doesn’t affect inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Search, so your Googlebot rules govern them. Applebot-Extended does a similar job for Apple’s foundation models.

So blocking training and blocking AI search are separate decisions: you can disallow GPTBot and ClaudeBot and still allow the search crawlers. Blocking those has a cost. OpenAI says sites that opt out of OAI-SearchBot won’t be shown in ChatGPT search answers, and Anthropic says blocking Claude-SearchBot may reduce your visibility, so you can drop out of those answers and their citations.

Two caveats. Some operators say their user-triggered fetchers may not follow robots.txt, because a person asked for the page. And CDN or firewall settings can block AI crawlers whatever the file says.

How to act on the results

Run the URLs that matter through the tester above: a page from each key template, the CSS and JavaScript files they load, and every section you mean to block, for Googlebot and for each AI crawler your policy covers. If a URL is blocked by mistake, fix the rule that caught it rather than piling exceptions on top.

Google generally caches robots.txt for up to 24 hours, so a change can take a day to register. After unblocking important pages or fixing a fetch error, request a recrawl in Search Console’s robots.txt report. OpenAI and Perplexity say their systems can take about a day to reflect changes too.

Finally, write your AI policy down, with the reasons, and review it when the site changes or an operator documents a new user agent. Keep the file short, commented and free of anything you wouldn’t publish.