● The ranking: Microsoft 76.04k, OpenAI 33.5k, Perplexity 31.09k, then Googlebot fourth on 24.64k.
● Answer engines vs Google: OpenAI, Perplexity, Anthropic and DuckAssist together made 78.68k requests against Googlebot’s 24.64k — more than three to one.
● The embarrassing one: our most-crawled URL was not an article. It was our own view counter, hit 14.8k times.
● And: 24,000 requests failed, roughly one in ten.
These are our own Cloudflare logs for aitoolsrecap.com over 30 days to 14 November 2026. We publish daily, which drives crawl frequency, so the absolute numbers are ours rather than universal. The ordering is the interesting part, and so is what we found when we looked at which URLs were being crawled.
The ranking
| Crawler |
Top agent |
Allowed requests, 30 days |
| Microsoft |
BingBot |
76.04k |
| OpenAI |
ChatGPT-User |
33.5k |
| Perplexity |
PerplexityBot |
31.09k |
| Google |
Googlebot |
24.64k |
| Amazon |
Amazonbot |
13.66k |
| Anthropic |
ClaudeBot |
10k |
| ByteDance |
Bytespider |
9.46k |
| Baidu |
Baidu |
5.58k |
| Apple |
Applebot |
5.52k |
| DuckDuckGo |
DuckAssistBot |
4.09k |
A definition, before anyone argues with the headline
Cloudflare files all of the above under “AI Crawlers”, Googlebot and BingBot included, on the reasoning that both now feed AI answers. That is defensible but it is not the only way to cut it. If you count only the pure answer engines — OpenAI, Perplexity, Anthropic and DuckAssist — they made 78.68k requests against Googlebot’s 24.64k. Either way Google is not first, and on the narrower definition it is outnumbered more than three to one.
Googlebot came fourth, and that is the smaller story
Google was also the only crawler in the top four to shrink. Over the most recent seven days it fell 2.71% while Perplexity rose 26.72% and Anthropic rose 31.99%. Bing grew 10.17% from a much larger base, and pulled 2.36 GB of HTML off the site in 30 days — more than every other crawler combined.
The Bing number is worth sitting with if you are deciding where to spend indexing effort. Bing is the source behind Microsoft Copilot, and it is crawling harder than anything else by a factor of two. Most sites still treat Bing Webmaster Tools as an afterthought.
ChatGPT-User is not an indexing crawler, and that is the number that matters
OpenAI’s largest agent over 30 days was not OAI-SearchBot, the one that builds the index. It was ChatGPT-User, which fires when somebody asks ChatGPT a question and it goes and reads a page live, in the moment, to answer them.
That is a different signal entirely. An indexing crawl means a search engine may cite you one day. A ChatGPT-User fetch means somebody asked a question and your page was chosen as the answer, 33,500 times in a month. Nobody separates these in public reporting, and they do not mean the same thing.
The mix shifts depending on the window you look at
Over 30 days OpenAI leads Perplexity and ChatGPT-User is OpenAI’s top agent. Over the most recent 7 days Perplexity leads OpenAI, and OAI-SearchBot is the top agent instead. A single snapshot will tell you whichever story it happens to contain, which is an argument for looking at both windows before writing anything down.
Our most-crawled URL was our own analytics beacon
The single most-crawled path on the site over 30 days was /hit.ashx — 14,800 successful requests. It is our view counter. It returns nothing a crawler can read, index or cite.
That is 6% of our entire crawl budget spent on a tracking endpoint. Over the most recent seven days the top path was a stylesheet instead, at 5,080 requests. Between them, the two most-crawled URLs on an AI tools publication were a beacon and a CSS file.
There is a second-order problem. If a view counter fires for crawler traffic, the view counts it produces are partly crawler traffic. We had already flagged our own figures as inconsistent with Search Console click volume and suspected inflation without knowing the mechanism. This is the mechanism.
What we are doing about it, and what you should check
Two fixes, and they are not alternatives. Server-side, do not increment a view count when the user agent is a known bot — that fixes the data regardless of whether a crawler honours robots.txt. Then disallow the beacon path in robots.txt to stop the crawl budget going there at all. Do not block CSS or JavaScript the same way: Google needs those to render a page, and a tracking endpoint is a different case entirely.
One in ten requests failed
Of 242,000 crawler requests over 30 days, 186,410 returned HTTP 200 and roughly 24,000 were unsuccessful. Over the most recent seven days that figure rose 278% against the previous period, which is the kind of jump that usually means something changed rather than something degraded slowly.
We do not yet have the split between 403, 404 and 5xx, and we are not going to guess at it in public. We will update this section when we have pulled it. One candidate worth naming for anyone else on IIS: long URLs can return 403 rather than 404 under default request filtering, which looks like a blocked crawler rather than a missing page and is easy to misread.
The general point stands without the breakdown. If one request in ten fails, a tenth of everything you publish may not be reaching the systems you are publishing for, and no amount of writing fixes that.
What to check in your own logs
All of the above came from one dashboard and took about ten minutes. If you have Cloudflare in front of your site, the same data is already there.
| Check |
What you are looking for |
| Top crawled paths |
Anything that is not content in the top five. Beacons, stylesheets, tracking endpoints and search pages are the usual offenders |
| Unsuccessful request rate |
Above a few per cent and something is wrong. Get the status split before theorising |
| Which agent, not just which company |
ChatGPT-User and OAI-SearchBot mean different things. So do ClaudeBot and Claude-User |
| Does your view counter fire for bots? |
If it does, your analytics are partly crawler traffic and any decision based on them is too |
| 7 days against 30 days |
The ranking changes. One window on its own will mislead you |
Frequently asked questions
Which AI crawler visits websites most?
On our logs over 30 days, Microsoft’s BingBot by a wide margin at 76.04k requests, then OpenAI at 33.5k, Perplexity at 31.09k and Googlebot fourth at 24.64k. Your ratios will differ with your publishing frequency and site size, but the ordering is worth checking rather than assuming.
Is Perplexity crawling more than Google?
On our site, yes. Perplexity made 31.09k requests over 30 days against Googlebot’s 24.64k, and over the most recent seven days Perplexity grew 26.72% while Google fell 2.71%.
What is the difference between ChatGPT-User and OAI-SearchBot?
OAI-SearchBot builds the index. ChatGPT-User fetches a page live when a user asks ChatGPT something and the model goes to read it. The second is closer to a citation than a crawl, and it was OpenAI’s largest agent on our site over 30 days.
Should I block AI crawlers?
That depends on whether you want to be cited in AI answers. Blocking them removes you from the systems a growing share of readers now ask first. Blocking a tracking endpoint or an infinite search URL is a different decision from blocking an answer engine, and worth keeping separate.
Does crawl volume mean citations?
No. It means your pages were fetched. Whether they get quoted is a separate question, measured in a different place — Bing Webmaster Tools reports Copilot citations, for example. High crawl volume is a precondition, not a result.
Sources and method