METHOD
36 days of first-party server logs, 24 July to 28 August 2026, on a single AI-tools publication. Crawlers identified by user agent. Figures reported as indexed values against a 100 baseline for the week of 1 to 7 August.
● Sample: one site, one vertical. Directionally useful, not universal.
● Headline: one AI crawler fell to 31 on the index. Another rose to 242. Neither moved because the site changed.
Why measure this at all
Publishers have spent two years learning to read Search Console. Almost nobody watches their own logs for AI crawler behaviour, and the two tell different stories.
That gap became visible in August. Promptwatch tracking showed reddit.com holding a 3.83 percent share of ChatGPT Search citations through early August, then falling to 0.52 percent between the 14th and 17th — an 86 percent collapse. The trigger was a retrieval change on 8 August, when ChatGPT Search began using the site: operator at scale, with domain-scoped queries jumping from 0.37 percent to 16.8 percent of background searches in a single day.
That was third-party sampling of citations. We wanted to know what the same period looked like from inside a server log — not which domains got cited, but which crawlers actually arrived.
What we found
| Week |
ChatGPT |
Claude |
Bing |
Google |
| Jul 24–31 |
76 |
103 |
66 |
121 |
| Aug 1–7 (baseline) |
100 |
100 |
100 |
100 |
| Aug 8–14 |
79 |
335 |
87 |
92 |
| Aug 15–21 |
57 |
181 |
80 |
90 |
| Aug 22–28 |
31 |
242 |
80 |
110 |
THE CONTROL IS WHAT MAKES THIS READABLE
If the site had broken, changed its robots.txt, or dropped in quality, every crawler would have moved together. They did not.
Google stayed within 10 points of baseline across the whole period. Bing settled 20 points below and held flat. Claude more than doubled. Only one channel moved sharply, and it moved in one direction.
The timing
ChatGPT crawl volume sat at or above baseline through the first week of August. The decline begins in the week starting 8 August — the same week the retrieval change was observed — and continues for three consecutive weeks without recovering.
We cannot prove causation from one site's logs. What we can say is that the inflection point falls in the same week as a documented change in how ChatGPT Search retrieves, that no other crawler shows the same pattern, and that nothing changed on the site to explain it.
The mechanism is at least consistent. If a retrieval system shifts from searching the open web toward querying named domains directly, the effect on any individual site depends entirely on whether that site is one of the domains it names. Some sites should see crawl volume rise. Most should see it fall.
The finding that generalises
Reddit lost 86 percent of its ChatGPT citation share in three days. We observed a 69 percent decline in ChatGPT crawl volume across three weeks. Different metrics, different sites, same direction, same trigger window.
And this is the second time in under a year. In September 2025 Reddit's ChatGPT citation share fell from 29.2 percent to 5.3 percent after Google removed the num=100 search parameter that OpenAI's data suppliers relied on. Neither time did Reddit change anything.
AI CRAWL SHARE IS A LIVE SIGNAL, NOT AN ASSET
A backlink profile erodes over months and mostly reflects work you did. AI crawl share can halve in a fortnight because a vendor changed a retrieval default nobody announced.
Anyone treating citation counts or AI referral volume as an accumulating asset is measuring something that behaves like a ranking position — and reporting it without a date attached.
The other half: diversification happened on its own
The more interesting result is what did not happen. Total crawler activity did not fall proportionally, because the composition changed.
- Claude crawling rose to 242 on the index, more than doubling from baseline and peaking higher still mid-month.
- Bing held flat at 80 across three consecutive weeks — the steadiest channel in the dataset.
- Google barely moved, ending 10 points above baseline.
- Referral traffic recovered, ending the period above where it began in August.
A publisher watching only ChatGPT would read a crisis. A publisher watching the whole crawler mix would read a reallocation, with one channel contracting while another expanded faster than it contracted.
That is the practical argument for measuring all of them rather than the one everybody talks about.
What to do with this
| If you publish anything |
Do this |
| Log AI crawlers by user agent |
Search Console will not show you this. Your own logs will, and they move weeks before citations do |
| Track them as an index, not a total |
Absolute counts tell you little. Relative movement against a baseline tells you when something changed |
| Watch the others as a control |
One crawler moving is a vendor change. All of them moving is your problem |
| Date every citation figure you quote |
A number that can move 69 percent in three weeks is a snapshot, and should be labelled as one |
| Do not optimise for one assistant |
The channel that grew fastest here is not the one most publishers are tuning for |
Limitations
Stated plainly, because a study that hides these is not a study.
- One site, one vertical. An AI-tools publication is not representative of news, e-commerce or documentation. Direction is more transferable than magnitude.
- User-agent identification is imperfect. Agents can be spoofed, and some traffic is unclassifiable. We report what identifies itself.
- Correlation, not causation. The inflection falls in the same week as a documented retrieval change. That is suggestive. It is not proof.
- 36 days is short. Long enough to see a step change, too short to separate trend from seasonality.
- Crawl volume is not citation volume. They are related and they are not the same measurement.
We are continuing to log and will publish an update if the pattern holds, reverses, or repeats on a different channel.
FAQ
What did the study measure?
AI crawler requests to a single publication over 36 days, from 24 July to 28 August 2026, identified by user agent and reported as indexed values against the week of 1 to 7 August.
Why did ChatGPT crawl volume fall?
The decline begins in the week of 8 August, when ChatGPT Search was independently observed shifting toward domain-scoped site: queries. We can show the timing correlation, not the cause.
Did other crawlers fall too?
No, and that is the important part. Google stayed within 10 points of baseline, Bing settled 20 points below and held flat, and Claude more than doubled. A site-side problem would have moved all of them.
Does this mean AI referral traffic is unreliable?
It means it behaves like a ranking signal rather than an accumulating asset. It can change substantially in a fortnight because of a vendor decision, in either direction.
How do I track this on my own site?
Log requests by user agent and group the AI crawlers separately. Track each as an index against a baseline week rather than as raw counts, and watch the others as a control.
Is the sample big enough?
For observing a step change across four weeks, yes. For estimating industry-wide magnitude, no. Treat the direction as informative and the numbers as one site's experience.