THE VERDICT
● Block everything: defensible if your content is your product and AI referrals are worth nothing to you.
● Allow everything: maximum reach, no leverage, your work trains competitors.
● Block training, allow search: what most sites should pick, and what Cloudflare now does by default.
What changed on 15 September
Cloudflare set new defaults across three categories of AI traffic. Training and Agent crawlers are blocked by default. Search bots remain allowed. It applied to new customers, to new sites added by existing customers, and to existing free-plan customers who had not set a preference beforehand. Paid customers who had already configured settings were not changed.
If you are on a free plan and have never opened the AI settings, your configuration changed without you doing anything, and a drop in your logs dated 16 to 18 September is the most likely fingerprint.
The three positions
| Block all | Block training only | Allow all |
|---|
| Cited in AI answers | No | Yes | Yes |
| Used for training | No | No | Yes |
| Referral traffic | None | Retained | Retained |
| Server load | Lowest | Moderate | Highest |
| Licensing leverage later | Strongest | Partial | None |
THE TRAP IS THE WILDCARD
Most sites that lost AI visibility did not decide to. They wrote one blanket rule aimed at training crawlers, and it caught the search crawler too.
Name the bots individually. A rule that says block AI is not a policy, it is an accident waiting for a log review.
What we would tell a small publisher
- If AI referrals are a real channel for you, keep search crawlers open. There is no upside to blocking the thing that sends readers.
- If your content is the product - a database, a paid archive, original research you license - blocking training is reasonable and costs you very little.
- If you are hoping to be paid for training use, blocking is the only position that creates any leverage. Whether that leverage is worth anything at your scale is a separate question, and for most sites it is not.
- Check what you actually have before deciding. Test each agent and read the status code rather than trusting a dashboard summary.
Which one
| If you... | Pick |
|---|
| Run a content site funded by traffic | Block training, allow search |
| Sell access to the content itself | Block all. The referral is worth less than the product |
| Run docs or a developer site | Allow all. Being in the model is the point |
| Do not know what you are set to | Go and look. Since 15 September it may not be what you think |
FAQ
Did Cloudflare block AI crawlers on my site?
If you are on a free plan and had not set an AI crawler preference before 15 September 2026, the new default applies: Training and Agent crawlers blocked, Search bots allowed.
Does blocking AI crawlers affect Googlebot?
Ordinary search crawling is a separate category from AI crawlers. Check the rule you actually wrote rather than assuming, because a wildcard can catch more than you intended.
Is blocking training crawlers worth it?
It costs you little and gains you little unless you intend to license your content. The decision that actually moves traffic is what you do with the search crawler.