THE VERDICT
● GPTBot: training collection. Blocked by Cloudflare default since 15 September. Losing it costs you little commercially.
● OAI-SearchBot: this is the one that matters. It builds the index behind ChatGPT search. Still allowed by Cloudflare's new default.
● ChatGPT-User: live fetch, thinking mode only. Largely outside your control.
Side by side
| GPTBot | OAI-SearchBot | ChatGPT-User |
|---|
| Job | Collects training data | Builds the search index | Opens a page mid-session |
| Cloudflare default since 15 Sept | Blocked | Allowed | Blocked as an agent crawler |
| Blocking it costs you | Training inclusion only | Being findable in ChatGPT search | Fresh reads of changed pages |
| Industry trend since GPT-5 | Up roughly 2.9x | Up roughly 3.5x | Down about 28% |
| Fires for free users | Not user-triggered | Not user-triggered | No. Thinking mode only |
| Worth blocking? | Defensible choice | Almost never | Rarely worth it |
THE MISTAKE ALMOST EVERYONE MAKES
People block GPTBot because they object to training on their work, then wonder why ChatGPT stopped citing them.
Those are two different bots. Blocking GPTBot removes you from training data. It does not remove you from ChatGPT search. Blocking OAI-SearchBot is the one that makes you unfindable, and people do it accidentally with a blanket rule.
Why this is not academic
Our own server logs over 57 days show ChatGPT retrievals falling from a peak of 7,507 a day on 5 August to 1,118 on 18 September, an 85 percent drop. Over the same period OpenAI's crawl of the wider web roughly tripled.
Until the three agents are separated in the logs, that number cannot tell you whether you have a blocking problem or a standing problem, and those have opposite fixes. We had the same undifferentiated column until this week.
How to tell them apart
- Split the user agents in your access log and count each per day. If one dropped and the others held, you have your answer in five minutes.
- Test with a real user-agent string and read the status code. 200 means access. Cloudflare's dashboard bundles 403, 429 and 404 into a single unsuccessful-requests figure, which will mislead you.
- Check the date of the change against 15 September. A cliff there is the Cloudflare default, not your content.
- Check robots.txt for blanket rules. A disallow aimed at training will catch the search crawler too if it is written against a wildcard.
Which one to act on
| If you... | Do this |
|---|
| Want AI referral traffic | Allow OAI-SearchBot. Everything else is optional |
| Object to training on your work | Block GPTBot specifically. Do not use a wildcard |
| Saw a drop after 15 September | Check your Cloudflare AI settings first. Cheapest fix available |
| Saw a drop in early August | Not a blocking issue. Retrieval preferences changed |
FAQ
Does blocking GPTBot hurt my ChatGPT visibility?
No. GPTBot collects training data. OAI-SearchBot builds the index behind ChatGPT search, and that is the one that determines whether you can be surfaced and cited.
Which bot did Cloudflare block on 15 September?
Training and Agent crawlers were blocked by default; Search bots remain allowed. It applied to new sites and to existing free-plan customers still on default settings.
Why is ChatGPT-User so rare in my logs?
It only fires in thinking mode. Free instant mode does not open pages at all, which is a cost decision rather than a caching artefact.