Which AI crawlers are blocked most
Share of the 494 sites with a robots.txt that still let Googlebot in, where the bot may not fetch
/. Training crawlers lead; bots that fetch a page because a user asked
for it and AI search bots are blocked about half as often.
- Model training
- AI search
- User request
Scale 0–25%. Googlebot: 0%, Bingbot: 0.2%.
Show as a table
| Crawler | Operator | Purpose | Blocked | Named in robots.txt |
|---|---|---|---|---|
| Bytespider | ByteDance | Model training | 21.7% | 21.3% |
| CCBot | Common Crawl | Model training | 21.3% | 23.3% |
| GPTBot | OpenAI | Model training | 18.6% | 26.3% |
| ClaudeBot | Anthropic | Model training | 18.4% | 24.9% |
| Meta-ExternalAgent | Meta | Model training | 16.6% | 17.0% |
| Google-Extended | Model training | 15.4% | 22.3% | |
| Applebot-Extended | Apple | Model training | 15.2% | 17.2% |
| PerplexityBot | Perplexity | AI search | 14.2% | 23.7% |
| Amazonbot | Amazon | Model training | 13.8% | 15.0% |
| Perplexity-User | Perplexity | User request | 11.7% | 12.3% |
| Claude-User | Anthropic | User request | 10.7% | 13.8% |
| ChatGPT-User | OpenAI | User request | 10.5% | 19.4% |
| DuckAssistBot | DuckDuckGo | AI search | 10.3% | 10.5% |
| MistralAI-User | Mistral | User request | 10.3% | 7.7% |
| Claude-SearchBot | Anthropic | AI search | 9.9% | 13.6% |
| OAI-SearchBot | OpenAI | AI search | 8.5% | 17.4% |
"Don't train on us, but do cite us"
The most common deliberate policy is a split. OpenAI and Anthropic each run separate crawlers for model training (GPTBot, ClaudeBot) and for their AI search (OAI-SearchBot, Claude-SearchBot), and robots.txt can treat them differently. Of the 92 sites that block GPTBot, 50 still allow OAI-SearchBot; of the 91 that block ClaudeBot, 42 still allow Claude-SearchBot. Overall 56 sites (11.3%) block at least one training bot and none of the four AI search bots.
Sites that block GPTBot but allow OAI-SearchBot include yahoo.com, ebay.com, medium.com, forbes.com, reuters.com, canva.com, snapchat.com and flickr.com: content and commerce sites that want to show up when someone asks ChatGPT a question, without their pages becoming training data.
The forgotten Google-Extended
Google-Extended is not a crawler but a robots.txt token: Googlebot fetches the page either way, and the token decides whether Google may use it to train Gemini. 25 of the 92 sites that block GPTBot do not block Google-Extended, so the one AI company that already crawls everything for search can still train on them. If the intent is "no training", the rule has to name every training token, not just OpenAI's.
Newer bots are rarely named
GPTBot is named explicitly in 26.3% of robots.txt files, ClaudeBot in 24.9%. Claude-SearchBot appears in
only 13.6%, MistralAI-User in 7.7%. A bot that is not named falls back to the
User-agent: * group, so sites that wrote their AI rules a year ago
often allow the newer bots without having decided to.
Allowlists and partial rules
The 14 sites that block all 16 AI crawlers, including facebook.com, instagram.com, x.com, baidu.com,
msn.com, threads.com, dailymail.com, usatoday.com and yelp.com, mostly do it the other way round: they
disallow everyone with User-agent: * and then allow a short list of
search engines by name. Twenty sites in the sample work this way, so a new AI crawler is blocked there
from day one. reddit.com goes further: its robots.txt is a bare Disallow: /
for every crawler, Googlebot included, so it falls outside the base of this study.
At the other end, 48 sites name GPTBot but leave the homepage open: they disallow only some sections. yahoo.com, for example, lets OAI-SearchBot crawl the site except article pages. This study counts such sites as allowing the bot, so the real share of restricted content is higher than the chart shows.
The top 100 block more often
In the top 100, GPTBot is blocked by 24.5% of sites (12 of 49 with a robots.txt that allows Googlebot); from rank 101 to 1,000 the share is about 18%. Several of the very largest domains, google.com, microsoft.com, youtube.com, apple.com and bing.com, block none of the 16 bots; their owners run AI products of their own. wikipedia.org is open to all of them as well.
Methodology
- Domains: the Tranco list V354N of 1 November 2026, top 1,000.
- robots.txt fetched on 2 November 2026 from
https://<domain>/robots.txt, retried on the www host when the bare domain did not answer, following up to five redirects as RFC 9309 requires. - 706 domains answered; the other 294 are mostly CDN, DNS and API hosts without a website. 514 served a robots.txt, 189 had none (4xx or an HTML page) and 3 answered with a server error.
- The base for percentages is the 494 sites whose robots.txt allows Googlebot at the homepage. The 20 sites that block Googlebot too are mostly CDN and ad-tech hosts; reddit.com and xiaohongshu.com are among them.
- Each verdict is for the path
/and is evaluated by the same RFC 9309 parser that powers the AI Crawler Checker: the most specific user-agent group applies, the longest matching rule wins, Allow wins a tie. - robots.txt is a request, not an access control. Sites may also block bots at the firewall, which this study does not see.
Download the data (CSV, 1,000 rows) — one row per domain with the HTTP status of robots.txt and an allowed or blocked verdict for each of the 18 bots. Free to reuse under CC BY 4.0 with a link to this page.