Verifying crawler identity on an independent website

Owning a small website makes it possible to observe discovery directly instead of relying on an indexing service's claims. It also makes a subtle counting mistake visible: a request can say Googlebot without establishing that Google's mobile crawler visited. A short experiment on an isolated site produced exactly that case.

Two labels, no authenticated crawler

An ordinary free security scan of an owned, benign page sent several browser identities. Alongside old Android, iPhone, iPad and desktop browser strings, two requests used the entire user-agent string Googlebot:

2026-10-16T05:25:37+00:00  GET /  200
2026-10-16T05:25:38+00:00  GET /  200
client: 2a01:4f8:1c1b:66c9::1
user-agent: Googlebot

These were restored client addresses, not the Cloudflare edges that transported the requests. A reverse lookup with getent hosts 2a01:4f8:1c1b:66c9::1 produced no record and exit status 2. The user agent also lacked Android and Mobile. Neither request was accepted as evidence of Googlebot Smartphone.

The scanner need not be doing anything improper. Testing multiple user agents can reveal content that changes by client. The problem is treating those labels as authentication.

A counting rule for self-hosted observations

1. Keep the exact host, path, request time, response status and user agent. A robots.txt request is different from retrieval of a particular article.
2. Establish the real client address. Behind a proxy, trust forwarded client headers only from that proxy's actual transport ranges, never arbitrary senders.
3. For a candidate common Google crawler, check the published address ranges and require reverse DNS followed by forward confirmation. A reverse hostname alone, or a lookalike domain suffix, is insufficient.
4. For a mobile-only experiment, also require the Android and Mobile tokens. A different legitimate Google fetcher would not answer the same question.

Keep publication and discovery separate

Publishing a link and receiving an accepted response from a service are observable events. They are not yet proof that a search crawler discovered the page. An independent website can retain both timestamps and measure the gap, while keeping an unpublished URL as a control.

The first lab observation window lasted about 58 minutes and ended at 05:59:21 UTC. It contained no user agent matching Googlebot/ and no verified mobile Googlebot retrieval. That is a bounded negative observation, not proof that later discovery is impossible or that any service guarantees a particular crawl time.

This note uses a normal h-entry, explicit author and publication date, and syndication link so the original remains on the owned site. It is factual research about operating an independent website, not an indexing service, ranking promotion or request for readers to fetch test URLs. No target is submitted through Google's testing or indexing products.