We checked robots.txt on 1,069 top websites: who blocks AI crawlers, and who still allows ChatGPT search

Everyone has an opinion on whether websites are blocking AI crawlers. We wanted the numbers, so on 4 November 2026 we downloaded the Tranco ranking of popular domains, took the top 2,000, fetched each site's robots.txt, and parsed the 1,069 that gave us a usable file for rules about 13 AI crawlers. We also checked whether each site serves an llms.txt. The per-site results are in a CSV you can download.

The short version

• About 1 in 5 sites blocks the big training crawlers: GPTBot (19.7%), ClaudeBot (18.8%) and CCBot (20.7%). 25.1% block at least one of six major training crawlers.
• Far fewer block the crawlers that decide whether you can appear in an AI answer: OAI-SearchBot (6.7%), Claude-SearchBot (8.1%) and PerplexityBot (11.6%). 12.3% block at least one of the three.
• Blocking GPTBot is not blocking ChatGPT search. Of the 211 sites that block GPTBot, 139 (65.9%) still allow OAI-SearchBot.
• Blocking is flat across rank. Sites ranked 1-100 block GPTBot at about the same rate as sites ranked 1,001-2,000.
• 16.6% of sites serve an llms.txt, and those sites are five times less likely to block GPTBot (4.5% versus 22.8%).
• 7.2% of sites already use the new Content-Signal robots.txt directive to state AI-use preferences.

Training crawlers: about one in five sites say no

These are the crawlers vendors say collect pages for model training. OpenAI says disallowing GPTBot indicates your content should not be used for training, and Anthropic says the same of ClaudeBot. Blocking tends to come as a package: of the 268 sites that block at least one of the six major training crawlers, 87% block two or more and 74% block three or more. That looks like a policy decision, not a one-off rule.

Search and answer crawlers: far fewer block them

Overall, 131 sites (12.3%) block at least one of the three search crawlers, and 63 sites (5.9%) block all three. Those are the sites that have chosen to be invisible to AI answers, not just to AI training. Keep in mind that the user-triggered fetchers are a special case: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Blocking those two in robots.txt may do less than it appears to.

Blocking GPTBot is not blocking ChatGPT search

This is the most useful finding for anyone writing a robots.txt. Of the 211 sites that block GPTBot, 139 (65.9%) do not block OAI-SearchBot. That matches how OpenAI documents the two: the settings are independent, so a site can opt out of training while staying eligible for ChatGPT search. The remaining 72 sites block both, and OpenAI says a site that opts out of OAI-SearchBot will not appear in ChatGPT search answers, though it can still show up as a navigational link.

If you want to be cited, that split is the policy to copy: block the training crawlers if you object to training, and leave the search and user agents alone. Our guide to which AI crawlers to allow has an example file.

The accidental blockers

Thirteen sites (1.2%) block GPTBot only because of a wildcard: a User-agent: * group with Disallow: /, alongside separate groups that let in search engines such as Googlebot. That is a legitimate way to write a strict policy, but it silently shuts out every AI crawler the author did not think to name, including the ones that drive citations. If your robots.txt is an allowlist, check that it names OAI-SearchBot, PerplexityBot and Claude-SearchBot on purpose.

Bigger sites do not block more

GPTBot blocking sits between 18% and 22% in every band. The one visible trend is that the very largest sites are likelier to block search crawlers too (24.4% in the top 100, against 10.3% in the 1,001-2,000 band), but only 45 sites in the top band had a readable robots.txt, so treat that as a hint, not a result. The sites blocking GPTBot include familiar names such as nytimes.com, bbc.com, reuters.com, cnn.com, amazon.com and medium.com.

A new directive: Content-Signal

77 sites (7.2%) already use a Content-Signal line in robots.txt. It comes from Cloudflare's Content Signals Policy and lets a site say, per use, whether it permits search, ai-input (answers and grounding) and ai-train. Cloudflare describes the signals as expressing preferences, not as technical countermeasures, and notes some companies may ignore them. We did not count Content-Signal lines as blocks, because they do not instruct a crawler to stay away.

llms.txt: adopted by the sites that want to be found

177 of the 1,069 sites (16.6%) serve an llms.txt. Adoption was highest in the 101-500 band (23.9%) and lowest in the 1,001-2,000 band (12.3%). Sites with an llms.txt block GPTBot far less often than the rest (4.5% against 22.8%), which fits the file being adopted mostly by developer-tools, SaaS and documentation sites that want AI visibility. That is selection, not proof that llms.txt works. In fact, an Ahrefs study of 137,210 domains found that 97% of llms.txt files received no requests in May 2026. We cover this in llms.txt explained.

What to do with this

1. Decide training and search separately. Blocking GPTBot, ClaudeBot or Google-Extended is a training decision. Blocking OAI-SearchBot, PerplexityBot or Claude-SearchBot removes you from answers.
2. If your robots.txt is an allowlist behind User-agent: *, add the search crawlers explicitly.
3. Check your CDN, not just robots.txt. 137 of the 2,000 sites answered our request for robots.txt with 403 Forbidden even though we sent an ordinary browser user agent. A firewall or bot-protection rule can stop a crawler before it ever reads your file.
4. Test the file with a real parser. A rule that lands in the wrong group applies the wrong policy without any error. The free audit on the home page shows which AI crawlers your robots.txt currently blocks.

Method and limitations

• Sample: the first 2,000 domains of the Tranco top-1M list, downloaded on 4 November 2026. Many top domains are not websites (CDN, DNS and ad-network hostnames), which is why only about half produced a usable robots.txt.
• Collection: for each domain we requested https://domain/robots.txt, then https://www.domain/robots.txt, with a standard browser user agent. Outcomes: 1,243 returned HTTP 200, of which 1,104 were genuine robots.txt files (the rest were HTML pages). 495 gave no response at all (typical of CDN, DNS and ad hostnames, though some were timeouts), 137 returned 403 and 94 returned 404.
• Exclusions: 35 hosts whose robots.txt blocks Googlebot as well are APIs, ad servers or CDNs rather than publishers making an AI-specific choice, so we excluded them. That leaves 1,069 sites.
• Parsing: we follow RFC 9309. A crawler uses its own group if one exists and the * group otherwise, and counts as blocked only when the result is a whole-site Disallow: / with no allow for the root. Partial blocks such as Disallow: /private/ do not count. Python's standard-library parser disagreed with ours on about 2% of sites, mostly because it ends a group at a blank line, which RFC 9309 does not.
• llms.txt: counted when /llms.txt returned HTTP 200, was not HTML, and began with a markdown heading. 96 sites returned an HTML page with a 200 status and were not counted.
• Robots.txt shows intent, not enforcement. A crawler can ignore it, and firewall rules are invisible in it.
• Bias: popular sites skew toward large companies, and the sites that blocked our scripted request are probably likelier than average to block crawlers, so these figures may understate blocking among the biggest sites. They say little about small sites.

Frequently asked questions