Publishers block Anthropic's search crawler far more than OpenAI's

OAI-SearchBot and Claude-SearchBot do the same job: they fetch pages so an assistant can cite them in an answer. Neither trains a model. Among the sites that block AI crawlers by name, OpenAI's is allowed through by 85% and Anthropic's by 31%.

85%

allow OAI-SearchBot
11 of 13 sites

31%

allow Claude-SearchBot
4 of 13 sites

6 vs 1

sites blocking the training crawler while allowing the search crawler: OpenAI 6, Anthropic 1

That last number is the finding. Six sites made a considered decision about OpenAI: stay out of the training set, stay in the answers. Exactly one site made the same decision about Anthropic. This does not look like a judgement about Anthropic. It looks like an information gap: publishers have learned that OpenAI runs two crawlers with different jobs, and have not learned it about Anthropic.

A plate showing requests arriving at the edge
of a site, some served and some refused before they reach the page. — Every refusal here happens in front of the site, before robots.txt is ever consulted. None of it appears in the file most tools read.

The 14 sites we threw away, and why that matters

This is the part most crawler studies skip, and it is where the number comes from.

Of 131 domains, 27 refused at least one crawler while serving a plain control request. The obvious move is to report 27. But 14 of those 27 also refused our Googlebot, and nobody blocks Google. What those sites are actually doing is checking crawler identity: Cloudflare's verified-bot handling and its equivalents confirm the request came from the operator's network, and refuse anything merely claiming a crawler's name from an address that is not theirs.

Our request is exactly that, so it is refused, and the real crawler is waved through. Those 14 sites cannot tell us anything about AI blocking and are excluded. The headline figures use the remaining 13 sites that refuse AI crawlers while serving the search crawlers, which is the only combination that demonstrates a rule keyed on the name.

If you are running a crawler study, this is the trap. Counting every 403 as a block conflates "this site refuses AI crawlers" with "this site refuses unverified clients." The second is far more common and says nothing about AI policy. Send a control request that identifies itself honestly, and treat a site that also refuses your Googlebot as unmeasurable rather than as evidence.

What got refused, in the 13 sites that key on the name

Crawler	Purpose	Sites	%
ClaudeBot	training	9	69%
Claude-SearchBot	search	9	69%
GPTBot	training	8	62%
PerplexityBot	search	8	62%
OAI-SearchBot	search	2	15%

Read the last two rows together. Both are search crawlers. Both exist so an assistant can cite a page. One is refused by 15% of this set and the other by 69%.

The 13 sites, named

Domain	Refuses
nytimes.com	GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot
profound.so	GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot
theguardian.com	ClaudeBot, Claude-SearchBot, PerplexityBot
apnews.com	ClaudeBot, Claude-SearchBot, PerplexityBot
newyorker.com	ClaudeBot, Claude-SearchBot, PerplexityBot
vox.com	ClaudeBot, Claude-SearchBot, PerplexityBot
theverge.com	ClaudeBot, Claude-SearchBot, PerplexityBot
forbes.com	GPTBot, ClaudeBot, Claude-SearchBot
linear.app	GPTBot, Claude-SearchBot, PerplexityBot
patreon.com	GPTBot, ClaudeBot
time.com	GPTBot
spiegel.de	GPTBot
arstechnica.com	GPTBot

Note time.com, spiegel.de and arstechnica.com: GPTBot refused, every search crawler allowed. That is the AEO-literate configuration, and it is what the other ten could have.

What an AEO practitioner should take from this

Audit your client's robots.txt against their edge, not instead of it. None of the refusals above appear in robots.txt. A checker that reads that file will report all thirteen of these sites as fully open.

Separate training from search when you advise. A publisher who says "block the AI crawlers" usually means "do not train on me." That is achievable while staying citable, and on current evidence most sites doing it for OpenAI have not done it for Anthropic. If your client wants to be in Claude's answers, Claude-SearchBot is a separate decision from ClaudeBot and probably needs raising explicitly.

Check whether the block was a decision at all. Several of these are CDN defaults nobody chose. The question to ask a client is not "should we allow AI crawlers" but "did you know you are refusing these, and did you mean to?"

Do not trust your own spoofed fetch. Ours is an indication, not evidence, and we say so. A firewall keying on address or forward-confirmed reverse DNS can refuse you and admit the real crawler, or the reverse. Server logs settle it.

Method, and what it cannot tell you

One request per crawler user agent, plus a control request that identifies itself honestly as ours, all issued in the same call so every site meets the same network conditions in the same moment. 131 domains, chosen to be nameable and spread across publishers, SaaS, developer tools, ecommerce and agencies. The list ships with the data: a study over a sample nobody can see is an anecdote.

Bucket	Sites	Meaning
served everything	84	no refusals at all
keys on the name	27	refused a crawler while serving the control
of which usable	13	also served Googlebot, so the rule is about AI
of which excluded	14	also refused Googlebot: verified-bot enforcement, unmeasurable from outside
inconclusive	16	the control was refused too, so the site turns away anything that is not a browser
unreachable	4	could not be audited at all

The 16 inconclusive sites are named in the data and include Reuters, the FT, the Economist and Bloomberg. We cannot say whether they single out AI crawlers, and we do not. Reporting the sites we could not conclude about is what should make the thirteen believable.

Sample size. Thirteen sites is a first data point, not a law. It is stated here rather than buried because the same list will be re-run monthly, and the trend is the part worth trusting. If you want a bigger sample, the tool that produced this is free.

Reproduce it

The data, the domain list and the code are all published. Disagreeing with this should cost you about four minutes.

npx openaeo-audit nytimes.com          # one site, in your terminal
git clone https://github.com/charlacsina/openaeo-audit
node tools/crawler-study.js tools/study/web-500.txt

Raw dataset (JSON) · the rubric as a specification · the same method on one site, explained

Why we can publish this and most tools cannot. Every check in our rubric declares what evidence it may be scored from, and anything we could not measure reports as not-measured rather than scoring zero. That rule is what forced the 14 exclusions above. A tool that scores what it did not measure would have published 27 and been wrong by a factor of two. Run it on your own site, free, no signup.