CrawlglassBot
If you found this address in your server logs, this is us. Crawlglass is a tool that fetches a page the way an AI assistant’s crawler does and shows the site’s owner what the bot could and could not read. Everything below is what the code actually does, including the parts that are awkward.
What it sends
CrawlglassBot/1.0 (+https://crawlglass.com/bot)
We never pretend to be a browser or another company’s crawler. Every request is a plain GET over http or https, and we never run your JavaScript — that is the whole point of the product.
Why we fetched your page
Somebody asked us to. Every fetch is started by a person typing an address into Crawlglass, not by a schedule of our own. We do not maintain a crawl frontier, we do not index you, and we do not come back on our own afterwards.
How robots.txt is treated — the honest version
A whole-site scan obeys it. The one address the customer pastes is fetched first, because we need your real origin after redirects before we can read the right robots.txt. From then on every page we discover — from your sitemap or from your own links — is checked against your robots.txt for CrawlglassBot, and anything you disallow is never fetched. The sitemap files themselves we do fetch: you named them in robots.txt. The report tells the customer how many of their own pages we were refused, so the rule is visible rather than silently routed around.
A single-page scan does not. When someone pastes one address, we fetch that one page without consulting robots.txt first — the same as a validator or a page-speed tool, and on the same reasoning: it is one page, fetched once, because a person explicitly asked for it. We think that is defensible, but we would rather you heard it from us than discovered it. If you disagree, the last section turns us off completely.
How hard it crawls
A single-page scan is three requests: the page, your robots.txt and your llms.txt. A whole-site scan reads up to 100 pages, five at a time, plus your robots.txt, your llms.txt and up to 10 sitemap files — 112 requests at most in the normal case, once, and then it stops. An address that turns out to redirect off your domain is fetched and then dropped without counting as one of the 100, so a site full of those costs us a few more than 112 — still a few hundred at the outside, and still only once. Each request gives up after 12 seconds, stops reading at 2 MB, and follows at most 5 redirects.
Where it comes from
Crawlglass runs on Vercel, so requests arrive from Vercel’s address ranges and we cannot offer you a fixed IP to allowlist. The user agent above is the reliable way to identify us.
What it will not do
We do not try to get past a login, a paywall or a bot wall — if a challenge page answers instead of your content, we report the wall and stop. We do not submit forms, we do not follow mailto: or tel: links, and a whole-site scan never wanders onto a domain that is not the one being scanned.
What we keep
Nothing, in the ordinary case. There is no database of scans: the result goes to the browser of the person who asked and we do not hold a copy. If a fetch fails, the address goes into our server log; an unexpected failure — not simply your site being slow or down — also goes to our error tracker, so the scanner can be fixed. One exception we publish deliberately: our scoreboard is a kept scan of 20 well-known home pages, re-run by hand and committed to our own source. If yours is on it and you would rather it were not, the email below works. The privacy page is the long version.
How to block it
To stop whole-site scans of your site, put this in your robots.txt:
User-agent: CrawlglassBot Disallow: /
We will read that and queue none of your pages: the one address somebody pasted is still fetched once, as said above, and then the scan stops with nothing it is allowed to crawl. It will not stop someone running a single-page scan of one address — to refuse us outright, block the user agent CrawlglassBot at your server, CDN or firewall, and we will report the block to whoever asked rather than work around it.
A person reads the replies
If CrawlglassBot did something it should not have, or you want it to stop and would rather just tell us, email mike@gearyworks.com. That reaches Mike, who built it and who will answer you himself.
Curious what we would say about your own site? Run a free scan. The limits of one are written down too.