FROM OUR OWN LOGS, 57 DAYS
● Peak: 7,507 ChatGPT retrievals on 5 August.
● Now: 1,118 on 18 September. Down 85%.
● First 14 days vs last 14: 5,464/day → 1,820/day, a 67% fall.
● Meanwhile: Perplexity +9%, Claude +60% over the same window.
● And: our robots.txt allows everything. Nothing was blocked there.
First, what it is not
It is not OpenAI crawling less. Across the wider web, OAI-SearchBot activity has risen roughly 3.5x and GPTBot roughly 2.9x since the GPT-5 launch. The web is being crawled more. We are being retrieved less.
It is not robots.txt. Ours is User-agent: * and Allow: / with a sitemap and no disallow lines.
And it is not a smooth decay, which is the most useful thing in the data. The curve has two cliffs with flat stretches between them. Gradual decline suggests content going stale. Step functions suggest somebody changed a setting.
Cliff one: 8 August
| Date |
ChatGPT retrievals |
| 5 Aug | 7,507 (peak) |
| 10 Aug | 7,021 |
| 14 Aug | 3,783 |
| 22 Aug | 1,921 — a 73% fall from peak |
On 8 August, ChatGPT Search's use of the site: operator inside its fan-out queries jumped from about 0.37 percent to 16.8 percent in a single day. Instead of searching the open web and choosing among results, it began scoping queries to specific domains — asking what a particular site says about a topic — and favouring official brand sites in what it cited.
THE CORROBORATING NUMBER IS REDDIT
Between 8 and 17 August, Reddit's share of ChatGPT citations fell from a 3.83 percent average to 0.52 percent — an 86 percent relative drop.
Reddit and a tools directory are the same kind of source: third parties writing about other people's products. When retrieval scopes to the vendor's own domain, that is exactly the category that gets skipped. Our 85 percent and Reddit's 86 percent are the same event.
A related contraction happened just before it. Between late July and early August, fan-outs in paid Thinking mode fell from 3.56 to 1.90 per conversation, and URLs touched fell from 47.2 to 33.9. Fewer pages per answer, then a preference shift in which pages.
Cliff two: 15 September
On 15 September, Cloudflare changed its AI-crawler defaults. Training and Agent crawlers are now blocked by default; Search bots remain allowed. It applied to new sites and to existing free-plan customers still on default settings who had not set a preference beforehand.
| Date | ChatGPT retrievals |
|---|
| 16 Sept | 1,907 |
| 17 Sept | 1,382 |
| 18 Sept | 1,118 — 41% in two days |
If you are on Cloudflare and never opened the AI settings, this is the one you can undo this afternoon.
The third factor, which nobody can fix
Live page fetches only happen in thinking mode. Free instant mode does not open pages at all — that is an economics decision at OpenAI, not a caching artefact. Retrieval now runs mostly off a discovery index and a shared reading cache, and the live fetch is reserved for the mode where users pay and wait.
Industry-wide, live fetches fell about 28 percent between December 2025 and March 2026, before any of the above.
Check which bot you actually lost
This is the part most dashboards get wrong, ours included until this week. "ChatGPT" in your logs is three different user agents doing three different jobs, and lumping them together hides the answer.
| User agent |
Job |
If this is what fell |
| GPTBot | Training crawl | Cloudflare now blocks it by default. Commercially it costs you little |
| OAI-SearchBot | Builds the ChatGPT search index | Still allowed by Cloudflare's new default. A fall here is a standing problem, not a blocking one |
| ChatGPT-User | Live fetch in thinking mode | Largely outside your control |
Split the column before you change anything. And test with a real user-agent string and read the status code directly — Cloudflare's own dashboard bundles blocked, rate-limited and not-found responses into one number, which will mislead you.
One more number worth your attention
Over the same 57 days our logs recorded 82,976 Bingbot hits and 12,927 Googlebot hits. Bing crawls us 6.4 times as often as Google does.
That is not a small asymmetry, and it goes a long way toward explaining why the same pages that Copilot cites constantly are close to invisible in Google. A page cannot rank on an index that rarely looks at it.
What we are doing about it
- Splitting the user agents. Until the three bots are separated, nobody can tell a blocking problem from a standing problem.
- Checking the Cloudflare toggle. Cheapest possible fix if it applies.
- Accepting the August loss. Domain-scoped retrieval structurally disadvantages anyone writing about products they do not own. No configuration change reverses that.
- Writing more of what a vendor cannot. An official site can publish its own pricing. It cannot publish a tested head-to-head against a competitor, and it cannot publish this article.
The honest caveat
This is one site's logs. A single domain is an anecdote, not a study, and our mix skews toward AI tooling coverage, which is precisely the category most exposed to a shift toward official sources. Treat the dates as well-evidenced and the magnitudes as ours rather than yours.
If you run a site and have your own numbers across these dates, we would rather compare logs than guess. The two cliffs should show up in your data too, or the explanation is wrong.
Sources
FAQ
Why did ChatGPT traffic to my site drop?
Three causes stack. On 8 August ChatGPT Search began scoping queries to specific domains and favouring official brand sites. On 15 September Cloudflare began blocking Training and Agent crawlers by default. And live page fetches only occur in thinking mode, which OpenAI rations for cost reasons.
Is OpenAI crawling less overall?
No. OAI-SearchBot activity is up roughly 3.5x and GPTBot roughly 2.9x since the GPT-5 launch. If your retrievals fell while the web's rose, the cause is specific to your site or your category.
What changed at Cloudflare on 15 September 2026?
Training and Agent crawlers are blocked by default and Search bots remain allowed. It applied to new sites and to existing free-plan customers still on default settings.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot collects training data. OAI-SearchBot builds the index behind ChatGPT search. ChatGPT-User performs live fetches during a session, and only in thinking mode. They have different business consequences, so separate them in your logs before drawing conclusions.
Does blocking GPTBot hurt my ChatGPT visibility?
Blocking GPTBot removes you from training data, not from ChatGPT search. Blocking OAI-SearchBot is the one that affects whether you can be surfaced and cited.
Can I get the August drop back?
Not by changing settings. It reflects a retrieval preference for official sources. The content that still gets retrieved is the content a vendor cannot publish about itself — tested comparisons, head-to-head verdicts, and first-party data like this.