Crawlers

Baiduspider: user agent, robots.txt, and verification

Baiduspider discovers and refreshes pages for Baidu search. Baidu’s crawler documentation distinguishes ordinary search indexing from specialized agents; a broad match on the word Baidu would also catch unrelated software.

User agent and purpose

Documented purpose: Search.

Baiduspider (search crawler token; verify the full variant against Baidu’s reference)

Baidu’s official spider reference (Chinese) identifies the crawler family. Do not treat a browser or mobile app bearing the Baidu brand as Baiduspider.

Robots.txt control

User-agent: Baiduspider
Disallow: /

Baidu documents the Baiduspider token for restricting search crawling in its official robots help. This is a crawl preference, not a reliable way to remove an already indexed URL immediately. Specialized agents may have different behavior.

Review this example in the context of your existing groups and paths. Publish it at the origin’s /robots.txt and check the actual response. A policy change can take time to be read; it does not erase earlier copies of content.

Verify origin before trusting the name

Baidu’s official reference describes reverse DNS ending in .baidu.com or .baidu.jp. Check the domain boundary, not whether a hostname merely contains those words. We recommend a forward lookup back to the original address as an additional safeguard. This guide does not assert a current machine-readable IP feed.

A User-Agent is self-reported and can be spoofed. A matching header is a claimed identity, not an authenticated bot. Use the operator’s current IP or DNS procedure for origin confidence; those checks are not cryptographic request signatures and do not authorize access to private data.

What AnalyticFast detects today

AnalyticFast does not currently classify Baiduspider in its named crawler aggregate. This directory entry is research, not a claim of detection support. AnalyticFast does not authenticate bot identities or perform the IP/rDNS checks described here.

A useful next check

Keep search access decisions separate from bot-filtering rules for your human analytics. If Baidu is an intended acquisition source, evaluate actual referral visits and conversions; the crawler’s request count does not measure demand.

Official sources and review date

• Official Baiduspider purpose and reverse-DNS guidance
• Official English robots instructions

Last checked 2026-10-31. Verify the latest policy and network data before changing production access rules.

Back to the crawler directory.