Crawlers

Googlebot: user agent, robots.txt, and verification

Googlebot discovers and renders content for Google Search. Its browser-like desktop and mobile user agents are not evidence of a human visit. Google also operates other crawlers with different purposes and verification ranges.

User agent and purpose

Documented purpose: Search.

Googlebot/2.1 (token within documented desktop and smartphone User-Agent formats)

Google’s common crawler reference documents changing browser versions around the Googlebot token. An exact full browser-version match is too brittle for a lasting log filter.

Robots.txt control

User-agent: Googlebot
Disallow: /

This blocks Googlebot crawling and can harm Search discovery. For a training/usage preference, review Google-Extended separately. Robots.txt controls fetching; a URL can remain known to Search even when fetching is blocked.

Review this example in the context of your existing groups and paths. Publish it at the origin’s /robots.txt and check the actual response. A policy change can take time to be read; it does not erase earlier copies of content.

Verify origin before trusting the name

Follow Google’s verification guide: compare with the relevant published crawler IP ranges or use reverse DNS with forward confirmation. Common Googlebot hostnames include .googlebot.com and .geo.googlebot.com. Use the rules for the actual crawler category; a substring resembling Google in a hostname is insufficient.

A User-Agent is self-reported and can be spoofed. A matching header is a claimed identity, not an authenticated bot. Use the operator’s current IP or DNS procedure for origin confidence; those checks are not cryptographic request signatures and do not authorize access to private data.

What AnalyticFast detects today

AnalyticFast currently recognizes the self-reported Googlebot token on valid analytics ingest requests. Opt-in crawler aggregates count those requests only; they do not automatically observe every page fetch on your site. AnalyticFast does not authenticate bot identities or perform the IP/rDNS checks described here.

A useful next check

For a Search visibility problem, inspect robots access, the response code, and what the crawler can render. A valid crawler visit shows a fetch, not index inclusion or a ranking outcome. Compare customer visits and revenue in a separate report.

Official sources and review date

• Official Googlebot UA and robots token
• Official network verification

Last checked 2026-10-31. Verify the latest policy and network data before changing production access rules.

Back to the crawler directory.