Crawlers

GPTBot: user agent, robots.txt, and verification

GPTBot collects web content that OpenAI may use to train foundation models. A GPTBot request is not evidence that a person used ChatGPT, clicked a citation, or bought a product.

User agent and purpose

Documented purpose: Training.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

The documented request token is GPTBot. Versions can change. OpenAI’s crawler documentation distinguishes it from search and user-triggered fetchers.

Robots.txt control

User-agent: GPTBot
Disallow: /

This example opts out of GPTBot crawling for training. It does not opt out of OAI-SearchBot. Make the training and search decisions separately rather than blocking every OpenAI token together.

Review this example in the context of your existing groups and paths. Publish it at the origin’s /robots.txt and check the actual response. A policy change can take time to be read; it does not erase earlier copies of content.

Verify origin before trusting the name

Compare the actual connecting address with the current GPTBot IP feed. Do not use the searchbot or ChatGPT-User feed as a substitute. No reverse-DNS shortcut is asserted here.

A User-Agent is self-reported and can be spoofed. A matching header is a claimed identity, not an authenticated bot. Use the operator’s current IP or DNS procedure for origin confidence; those checks are not cryptographic request signatures and do not authorize access to private data.

What AnalyticFast detects today

AnalyticFast currently recognizes the self-reported GPTBot token on valid analytics ingest requests. Opt-in crawler aggregates count those requests only; they do not automatically observe every page fetch on your site. AnalyticFast does not authenticate bot identities or perform the IP/rDNS checks described here.

A useful next check

If training access is the concern, check the GPTBot group on each public hostname. If discovery in ChatGPT search is the concern, audit OAI-SearchBot separately. Keep the dates of policy changes so later log changes have context.

Official sources and review date

• OpenAI crawler purposes and robots controls

Last checked 2026-10-31. Verify the latest policy and network data before changing production access rules.

Back to the crawler directory.