If you run a site behind Cloudflare and you have ticked the box that blocks AI training crawlers, then from 15 September you will also be blocking Googlebot, Applebot and BingBot.

That is not a side effect anyone is hiding. It is stated plainly in Cloudflare's post of 1 July, by Jin-Hee Lee and Bryan Becker, which is worth revisiting now rather than in September because the window to pre-empt it closes then. Of multi-purpose crawlers — and it scopes this to "specifically those that combine Search with Training" — it says:

Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service).

The post is nearly four weeks old — it surfaced again this week — but the date that matters is ahead of it.

Why one checkbox now reaches search

Cloudflare has stopped classifying bots by what they are and started classifying them by what they do. The reasoning is that "AI" has become useless as a category: "We could debate the cutoff for what qualifies as 'AI' today, just to find that the standard changes tomorrow."

In its place are three behaviours every site owner can now manage, including on the free tier:

• Search — "any behavior that collects or indexes your content, so it can answer questions about it later." Cloudflare's position is that site owners "should expect to get referral traffic or other equitable compensation as a result."
• Agent — "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now", which covers chat fetch bots and browser-driving agents. "Often there's a human waiting on the other end."
• Training — "a crawler taking your content to train or fine-tune a model", where "your data is permanently absorbed into the underlying architecture of the AI to improve its capabilities."

The catch is that a single crawler can be more than one of these — and Cloudflare classifies Googlebot, Applebot and BingBot as combining Search with Training. Its framing is that multi-purpose bots were previously tracked under one purpose, and "should be tracked with all purposes, not just one of them"; that is the design principle already in force. What starts on 15 September is the enforcement: multi-purpose crawlers "will be allowed/blocked according to all of their behaviors", and, Cloudflare says, "the defaults will be enforced by the most restrictive applicable rules."

It is worth being clear where that classification comes from, because Google organises this differently — though not, it turns out, by disagreeing about what Googlebot does. Google's crawler documentation puts training under a separate robots.txt token, Google-Extended, which governs "whether content Google crawls from their sites may be used for training future generations of Gemini models" and which "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."

But Google-Extended is not a second crawler. It "doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." One crawler, two levers — and Cloudflare's classification of Googlebot as Search-plus-Training is consistent with that. Google's own entry adds that the same token covers "grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy)", so search-index content reaches Gemini under it too.

The difference is where the control sits. In robots.txt a publisher can refuse training and keep search, because the two preferences are separable after the fetch. Cloudflare enforces at the request — it blocks the crawler — and a single crawler cannot be half-blocked. That is precisely why Cloudflare wants the labs to split their automation into three crawlers, and why, until they do, blocking Training on Cloudflare costs you Googlebot.

Cloudflare is explicit that it wants the labs to fix this at their end: "If a company runs automation that builds Search indexes, acts as an Agent, and collects data to Train their models, then we strongly encourage that company to separate the automation into three separate crawlers." Until they do, the site owner absorbs the ambiguity.

The bargain, in Cloudflare's own words

The post is unusually candid about who this hurts:

If you run a small site, the problem isn't just that someone could train models on your content — it's that nobody can find you in the first place. So you have to make a Faustian bargain: either show up in search and let AI train on you, or risk losing discoverability.

And it names the structural consequence: this "unfairly advantages incumbent search providers if they use the same bots for both search and training; and this unfair advantage incentivizes new players to be evasive as they try to close the competitive gap."

That is a fair description of the position. It is also worth asking where the description comes from. Cloudflare sits between the crawlers and the sites, and the remedy it proposes — finer-grained controls, a bot database, a content-use setting — runs through Cloudflare. The diagnosis can be accurate and the vendor still be the beneficiary of it.

The other 15 September change

Separately, for new domains onboarding to Cloudflare from that date, Training and Agent will be blocked by default on pages that display ads, while Search stays allowed. The logic offered is that an ad "is a signal that a website owner meant for a person to land there and see it", so on those pages human attention is treated as the point.

That ad-page default is scoped to new domains; Cloudflare does not describe it as retroactive for existing zones. The multi-purpose rule is the one that reaches existing customers — the post names "customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service)", and the new controls are "live now, and can be configured by all existing customers in their zone Settings". Anyone who wants to keep training-and-search crawlers coming can say so "in their Security settings any time leading up to September 15", confirming they want "no changes on Training crawlers that also crawl for Search purposes"; Cloudflare says it will keep notifying customers before the date.

The rest of the taxonomy

The three configurable options sit inside a wider set of eleven behaviours Cloudflare now tracks: Search, Agent, Training, Transact (checkout on a user's behalf), Data Collection (price scraping, competitive intelligence, third-party analytics), Security Testing, SEO, Ads Verification, Social/Link Preview, Feed Fetching, and Monitoring & Operations. Only the first three are configurable by all customers today.

There is also a content-use dimension being built for Bot Management customers — what a crawler may keep and reshare — with three levels: immediate (interact, but store and reuse nothing), reference (index, excerpt and link back — the default), and full (summarize and reproduce).

Enterprise Bot Management customers get BotBase, a searchable database of tracked bots and their classifications on the dashboard. Cloudflare describes this as tackling visibility first, with a direct control centre promised "later this year".

Why this matters beyond Cloudflare customers

The Debian general resolution we covered this morning contains, in its strictest proposal, a description of what crawler load does to a volunteer project: scraping "effectively a large scale and perpetual Denial of Service attack on sites that many users rely on", with parts of the infrastructure unreachable and JS-based checks forced on as a defence. Debian is arguing about whether its developers may use these models. Cloudflare is selling the shield against the crawlers that feed them.

The two are the same problem seen from opposite ends, and only one of them has a checkbox. What the September change makes concrete is that the checkbox is not free: on current crawler design, the price of refusing to feed the training run is being harder to find.