Googlebot gets a 200. ClaudeBot gets a 403. Same page, same second.

The door was shut before it saw anything. GPTBot asked for the Times homepage and got a refusal, not a page. Nothing it might have learned from that page can reach ChatGPT, because it never received a word of it.

This refusal comes from the network in front of the website, not from the website itself. That matters because the usual place people look, the robots.txt file, does not mention any of this. Reading that file would tell you GPTBot is welcome.

This is the crawler behind ChatGPT's ability to look something up and cite it. Refused as well, which closes the second of the two routes into that assistant.

Blocking the collector and blocking the searcher are separate decisions. Here both were made the same way.

Same refusal, different company. The pattern across these five is the point: it is not one crawler having a bad day, it is a rule about a category of visitor.

If you run a site and want the opposite outcome, this is the layer you have to look at. No amount of tidying your pages changes a refusal that happens before the page is ever consulted.

Refused. This is the crawler that would fetch a page in order to quote it in an answer, so a refusal here is the difference between being cited and being absent.

Refused, completing the set. Five of the eight crawlers we ask for were turned away, and all five belong to assistants that answer questions rather than return a list of links.

Served the full page, immediately. This is what makes the contrast legible: the site is not closed, it is closed to a particular kind of visitor.

Search sends people to you. Answer engines often answer instead. Whichever side of that you want to be on, it should be a choice you made rather than one your network settings made for you.

Served. Worth sitting with, because Bingbot's crawl feeds Copilot. The same company's assistant gets in through the search door while the assistants without a search business are refused at it.

Served. Apple's crawler has been fetching pages for Siri and Spotlight for years, long before the current wave, and it is generally treated as part of the furniture.

This one is not a crawler. It is us, asking plainly, announcing nothing except our own name. It was served.

That single result is what turns the five refusals into a finding. If this request had also been refused, the honest conclusion would only be that the Times turns away anything unfamiliar, and we would have no business saying AI crawlers were singled out. Because it was served, we know the rule keys on the crawler's name. When the control comes back refused, this audit says inconclusive and stops there.

Served, of course. Nothing here is broken and nothing here is an outage. A human reader gets the page every time.

That is precisely why this is invisible without measuring it. Everything looks fine from a browser, and the only visitors being refused are the ones nobody watches.