Twenty well-known sites, fetched the way an AI crawler fetches them

3 of them never let us in at all

GoDaddy, Canva and OpenAI turned the request away — 403. Not a rule in their robots.txt — a bot wall, the kind Cloudflare, DataDome and PerimeterX sell, deciding that an unfamiliar crawler does not get the page.

What that does not prove: those walls usually keep a list of verified crawlers they wave through, and OpenAI’s and Google’s bots are normally on it. So this is not us catching GoDaddy hiding from ChatGPT — and it is why every column after their name is a dash rather than a bad mark. What it does prove is that being readable is now a setting somebody has to get right, and the default for anything not on the list is no. If your site sits behind the same kind of wall, the question is whether the list it uses is the current one.

The ones that opened the door were all readable

This surprised us, and it is worth saying plainly rather than burying: not one of the 17 sites that answered failed the check for readable text without JavaScript. Every one of them served the crawler real words in the HTML itself. The spread is wide — Squarespace hands over 3,002 words of its home page and Notion hands over 258 — but at this size that is mostly a difference in how much a company chooses to say on its front page.

The most common failing check on this table is duller than any of that. On 6 of the 17 that answered, fewer than half the images carry alt text. A crawler cannot see a picture. Alt text is the only part of one it can quote.

13 publish an llms.txt. 4 publish no schema at all.

An llms.txt is a plain-text file telling an assistant what a site is and which pages matter. Schema.org markup is the machine-readable version of “who we are, what we sell, how to reach us.” The odd part is which one is further along: the convention that barely existed a couple of years ago is on 13 of the 17 sites we could read, while 4 of them still carry no schema markup at all. Those are the two files Crawlglass generates for you, which is not a coincidence — they are the two an assistant actually reads.

Exactly one site here turns a named AI agent away in robots.txt on purpose: Figma. Everything else on this table is an accident. That one is a decision somebody made.

How to read this fairly

• It is one page — the home page — on one day. A low grade here is not a verdict on a whole site.
• It measures what a crawler receives, not what any assistant decides to say about the company. Nobody can measure that from outside, and we say so on what Crawlglass cannot do.
• We asked once, as CrawlglassBot/1.0 (+https://crawlglass.com/bot) — our own name, never somebody else’s. We did not retry, and we did not pretend to be a browser or another company’s bot to get past anything.
• The scan is deterministic code, not a language model judging anyone. You can run the identical scan on any of these sites and get this table back.