Crawlers

Bingbot: user agent, robots.txt, and verification

Bingbot finds new and updated pages for Bing’s index. A request bearing bingbot/2.0 is a crawler claim, not a search click or proof that the page appears in a particular result.

User agent and purpose

Documented purpose: Search.

Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

Microsoft’s crawler reference also documents rendering-oriented desktop and mobile formats. Keep the token check separate from browser version matching.

Robots.txt control

User-agent: bingbot
Disallow: /

This restricts Bingbot fetching. Microsoft’s crawl-control guidance documents robots directives and Crawl-delay. If the objective is simply reducing load, review crawl controls before excluding all search crawling.

Review this example in the context of your existing groups and paths. Publish it at the origin’s /robots.txt and check the actual response. A policy change can take time to be read; it does not erase earlier copies of content.

Verify origin before trusting the name

Microsoft’s verification guide requires reverse DNS ending in .search.msn.com and a forward lookup back to the same address. It also links a current Bingbot IP list; refresh rather than baking addresses into a permanent rule.

A User-Agent is self-reported and can be spoofed. A matching header is a claimed identity, not an authenticated bot. Use the operator’s current IP or DNS procedure for origin confidence; those checks are not cryptographic request signatures and do not authorize access to private data.

What AnalyticFast detects today

AnalyticFast currently recognizes the self-reported Bingbot token on valid analytics ingest requests. Opt-in crawler aggregates count those requests only; they do not automatically observe every page fetch on your site. AnalyticFast does not authenticate bot identities or perform the IP/rDNS checks described here.

A useful next check

Verify the source before diagnosing an apparent Bingbot traffic spike. Then inspect crawl controls and response status. Use a small rule change and compare subsequent server logs instead of assuming a user-agent restriction proves origin or compliance.

Official sources and review date

• Official Bingbot origin verification
• Official crawler formats
• Official robots and crawl controls

Last checked 2026-10-31. Verify the latest policy and network data before changing production access rules.

Back to the crawler directory.