robots.txt for AI Crawlers: Rules, Errors and Access

A 30-year-old draft running the internet

robots.txt was proposed in 1994 after a rogue crawler caused a denial-of-service on its author's own server. Early search engines adopted it within months, and for nearly three decades it stayed a de facto convention with no official rule book — practitioners simply guessed at the edge cases. That changed when Google, sitting on data covering 500 million websites and 4 billion host names, pushed the format through the IETF standards process: a draft submitted in 2019, iterated through 2020–2021, and published as RFC 9309 in September 2022.

That history matters for one reason: robots.txt is no longer folklore. It is a documented standard with defined behaviour for every response code, every encoding quirk and every origin — and the edge cases are exactly where money gets won and lost.

How Google handles robots.txt errors

For Google, a successful response supplies crawl rules. A 4xx response other than 429 means no robots.txt crawl restrictions; it does not grant access through authentication. Google handles 429 as a server error even though it is not a 5xx status. For 5xx errors, Google initially pauses crawling for 12 hours while retrying, then uses the last good file for up to 30 days if available. Without a cached file, it assumes no crawl restrictions during that phase.

After that period, behaviour depends on the website’s general availability. This is not a fixed ten-day deindexing rule. Monitor the endpoint, restore a valid response and check server logs and Search Console. The authoritative reference is Google’s robots.txt specification (checked 4 September 2026). Other crawlers may behave differently.

Crawl permission, indexing and access control are different things. A robots.txt rule is not a password or an instruction to remove a URL from search.

Google normally caches robots.txt for up to 24 hours, sometimes longer when refreshes fail; this is not a mandatory 24-hour delay for every change. Google parses up to 500 KiB. Keep the file small, test the response on every origin and protect private content with authentication, not Disallow rules.

The mistakes that silently break your file

Beyond status codes, most robots.txt failures are small syntax errors that quietly void the whole policy. The recurring ones worth checking today:

• Rules that don't start with / or *. A Disallow value without one is ignored outright.
• Domain names used as user agents. You block by user-agent string, not by the site scraping you.
• The host name added to a path. Disallow: https://example.com/private/ actually allows everything — paths are relative to the file's own origin.
• CSS and JavaScript blocked by accident. Search engines need to render the page; blocking rendering resources hurts more than it protects.
• Googlebot-only patterns. A short rule that allows only Googlebot and disallows everyone else by default also kills Bing traffic and every LLM crawler at once.
• A missing file on cloud storage. If a storage origin returns 403 for robots.txt, Google assumes no robots.txt restrictions. Actual object access still depends on permissions; robots.txt does not make a private bucket public.
• An invisible byte order mark. RFC 9309 requires UTF-8, but Windows text editors can silently prepend a BOM character to the file. If that mangles your first User-agent line, the whole group — and potentially the entire file — is ignored.

Two placement rules trip up large organisations specifically. robots.txt must sit at the root of an origin, never a subdirectory, and it is per origin — every subdomain, port and scheme needs its own file, because a robots.txt buried anywhere else is simply ignored, which means everything is allowed by default.

Crawl budget and the bot majority

robots.txt isn't just a defensive file anymore — it's how you shape an economy of crawlers competing for your server's attention. On 3 June 2026, Cloudflare Radar data showed automated requests now make up 57.5% of HTML traffic to web content, against 42.5% from humans — the first machine majority in internet history, arriving years ahead of Cloudflare's own forecast. Fastly's June 2026 analysis found AI request volume grew roughly 30% between January and May alone, about 6.5× faster than human traffic, with Claude-related requests up more than 555% over its January 2026 baseline.

One practitioner-reported Search Console breakdown found only 16% of crawl budget over 90 days went to HTML pages with a real chance of ranking, while 51% went to tracking pixels, fonts and other non-ranking file types — a single client's numbers, not an industry benchmark, but they show the shape of the waste.

Most of that crawling brings nothing back. Cloudflare tracked ClaudeBot crawling roughly 70,900 pages for every single referral it sent in mid-2025; even after improvements, the ratio stayed the worst of any major AI platform through the end of the year. If your store runs on Cloudflare, always disallow the challenge path — because each bot-check interstitial can spawn a unique URL and drown Googlebot in an infinite crawl trap it can't escape:

User-agent: *
Disallow: /cdn-cgi/

Wasted crawl budget is no longer just a Googlebot problem — it's a second internet of bots reading your pages, and robots.txt is the bluntest instrument most sites have to shape it.

Block training, keep the citations

The most useful 2026 development is that training and answer crawlers are now separate bots with separate user agents. GPTBot is not OAI-SearchBot. ClaudeBot is not Claude-SearchBot. That means you can block the crawler that feeds a model's training corpus while keeping the crawler allowed that might actually cite you in an answer:

User-agent: GPTBot          # OpenAI training — block to protect content
Disallow: /
User-agent: OAI-SearchBot   # OpenAI answers — allow to stay citable
Allow: /

Cloudflare has pushed this further with its Content Signals Policy — three machine-readable directives (search, ai-input, ai-train) that sit above your ordinary rules and separately permit indexing, inference and training. The IETF's AI Preferences working group is standardising the same idea as a vendor-neutral Content-Usage rule, with its attachment draft published in April 2026 and a target of delivering the full specification by 31 August 2026.

One caveat worth stating plainly: a Disallow rule is a request, not a wall. In 2025, Cloudflare caught Perplexity running undeclared "stealth" crawlers that ignored robots.txt entirely — a case now tangled in a federal DMCA lawsuit from Reddit. robots.txt declares your intent. CDN-level bot management is what actually stops the crawlers that don't respect it.