Is anything telling Google to stay away?

The line worth checking for. User-agent: * followed by Disallow: / tells every search engine to read nothing on the site. It is one line, it is often left behind after a rebuild, and a site can carry it for months while the owner wonders where the traffic went.

It is not always your web server that answers. We asked one live site for its /robots.txt twice, about an hour apart, from the same place, and got two different files. Once it was the site’s own — Disallow: /wp-admin/, perfectly normal. Once it was HTTP 200 text/plain containing Disallow: /, with Server: BitNinja Captcha Server on the response: the security product in front of the site answering on its behalf. At the same time the home page went from a 403 anti-robot page to the real 553-word article.

Nothing was wrong with the file the owner wrote. A crawler that arrives during one of those spells is simply handed a different one, and treats it as real. This check reads the Server header on the file itself, so when that happens it says who answered.

What this is not. A handful of fetches. It cannot tell you whether Google has actually indexed your pages — only Search Console knows that, and only for a site you have verified. It also cannot see a noindex tag, which lives on each page rather than in these files, and which blocks a page just as completely.

One answer, not every answer. We ask once, from one place, and report what came back. As the case above shows, a clean result is one good answer — not proof that every crawler, at every hour, gets the same one. If your traffic says otherwise, believe your traffic, run this again later, and look in Search Console for what Google was actually served.

Nothing is stored. The fetch happens from this site rather than your browser, and the domain you type is not saved or logged by us.