The Tiny File That Can Deindex Your Entire Site

A 30-year-old draft running the internet

robots.txt was proposed in 1994 after a rogue crawler caused a denial-of-service on its author's own server. Early search engines adopted it within months, and for nearly three decades it stayed a de facto convention with no official rule book — practitioners simply guessed at the edge cases. That changed when Google, sitting on data covering 500 million websites and 4 billion host names, pushed the format through the IETF standards process: a draft submitted in 2019, iterated through 2020–2021, and published as RFC 9309 in September 2022.

That history matters for one reason: robots.txt is no longer folklore. It is a documented standard with defined behaviour for every response code, every encoding quirk and every origin — and the edge cases are exactly where money gets won and lost.

The status code that can deindex you overnight

The single most consequential thing to understand about robots.txt is how asymmetrically it treats HTTP status codes. A 200 with parseable rules is the normal case. A 404, 410, 403 or 401 — anything in the 4xx range except 429 — is treated as no robots.txt at all, meaning everything is allowed. A 5xx response, including 429, is the opposite: the site is assumed "not open for business," so it's treated as fully disallowed.

That asymmetry is the trap. If your robots.txt endpoint returns a sustained 500 or 503, Google can't be sure whether a rule forbids crawling, so it proceeds with caution — and, over an extended period, may remove the site entirely. Practitioners have documented Googlebot halting all crawling after a string of 500 responses on robots.txt, with most pages deindexed roughly ten days later.

One little file not responding correctly with an HTTP status code can deindex your entire website.

Pair this with two details from the standard: bots may cache robots.txt for up to 24 hours, so a freshly added rule isn't honoured for a full day, and only the first 500 KB is guaranteed to be parsed — put your most important rules first. Then monitor the HTTP status of the robots.txt endpoint on every origin as a P1 alert. It is a more efficient way to vanish from Google than any algorithm update.

The mistakes that silently break your file

Beyond status codes, most robots.txt failures are small syntax errors that quietly void the whole policy. The recurring ones worth checking today:

• Rules that don't start with / or *. A Disallow value without one is ignored outright.
• Domain names used as user agents. You block by user-agent string, not by the site scraping you.
• The host name added to a path. Disallow: https://example.com/private/ actually allows everything — paths are relative to the file's own origin.
• CSS and JavaScript blocked by accident. Search engines need to render the page; blocking rendering resources hurts more than it protects.
• Googlebot-only patterns. A short rule that allows only Googlebot and disallows everyone else by default also kills Bing traffic and every LLM crawler at once.
• A missing file on cloud storage. An S3 bucket mapped to a subdomain with no robots.txt inside it returns a 403 — which means the whole bucket is open for crawling.
• An invisible byte order mark. RFC 9309 requires UTF-8, but Windows text editors can silently prepend a BOM character to the file. If that mangles your first User-agent line, the whole group — and potentially the entire file — is ignored.

Two placement rules trip up large organisations specifically. robots.txt must sit at the root of an origin, never a subdirectory, and it is per origin — every subdomain, port and scheme needs its own file, because a robots.txt buried anywhere else is simply ignored, which means everything is allowed by default.

Crawl budget and the bot majority

robots.txt isn't just a defensive file anymore — it's how you shape an economy of crawlers competing for your server's attention. On 3 June 2026, Cloudflare Radar data showed automated requests now make up 57.5% of HTML traffic to web content, against 42.5% from humans — the first machine majority in internet history, arriving years ahead of Cloudflare's own forecast. Fastly's June 2026 analysis found AI request volume grew roughly 30% between January and May alone, about 6.5× faster than human traffic, with Claude-related requests up more than 555% over its January 2026 baseline.

One practitioner-reported Search Console breakdown found only 16% of crawl budget over 90 days went to HTML pages with a real chance of ranking, while 51% went to tracking pixels, fonts and other non-ranking file types — a single client's numbers, not an industry benchmark, but they show the shape of the waste.

Most of that crawling brings nothing back. Cloudflare tracked ClaudeBot crawling roughly 70,900 pages for every single referral it sent in mid-2025; even after improvements, the ratio stayed the worst of any major AI platform through the end of the year. If your store runs on Cloudflare, always disallow the challenge path — because each bot-check interstitial can spawn a unique URL and drown Googlebot in an infinite crawl trap it can't escape:

User-agent: *
Disallow: /cdn-cgi/

Wasted crawl budget is no longer just a Googlebot problem — it's a second internet of bots reading your pages, and robots.txt is the bluntest instrument most sites have to shape it.

Block training, keep the citations

The most useful 2026 development is that training and answer crawlers are now separate bots with separate user agents. GPTBot is not OAI-SearchBot. ClaudeBot is not Claude-SearchBot. That means you can block the crawler that feeds a model's training corpus while keeping the crawler allowed that might actually cite you in an answer:

User-agent: GPTBot          # OpenAI training — block to protect content
Disallow: /
User-agent: OAI-SearchBot   # OpenAI answers — allow to stay citable
Allow: /

Cloudflare has pushed this further with its Content Signals Policy — three machine-readable directives (search, ai-input, ai-train) that sit above your ordinary rules and separately permit indexing, inference and training. The IETF's AI Preferences working group is standardising the same idea as a vendor-neutral Content-Usage rule, with its attachment draft published in April 2026 and a target of delivering the full specification by 31 August 2026.

One caveat worth stating plainly: a Disallow rule is a request, not a wall. In 2025, Cloudflare caught Perplexity running undeclared "stealth" crawlers that ignored robots.txt entirely — a case now tangled in a federal DMCA lawsuit from Reddit. robots.txt declares your intent. CDN-level bot management is what actually stops the crawlers that don't respect it.