The takeaway: If your site sits behind Cloudflare, you can now refuse AI training by Google's and Apple's crawlers without dropping out of their search results, with Bing due to follow. Cloudflare's new Disallow AI Training setting, announced on 15 September, keeps Googlebot, Applebot and Bingbot crawling for search while telling Google and Apple not to train on your content (Bing support is due in early 2027, see below). It also changes what the plain Block setting means, so it's worth a two-minute check.

This updates our July piece, which reported Cloudflare's earlier plan: from 15 September, a site blocking AI training would also block Googlebot, Applebot and Bingbot. Cloudflare has taken a different route.

The problem it solves

Some of the biggest crawlers are "mixed-use": one bot collects pages both for search and for AI training. Until now, refusing one meant refusing the other. Cloudflare's own figures show why that mattered: fewer than 1% of Cloudflare sites block search bots, but 17% use some mechanism to block training.

What the new setting does

Cloudflare's post, by Bryan Becker, says Disallow AI Training works like this:

• It publishes a no-training preference in your robots.txt through a feature called Bot Preference Sync.
• Mixed-use crawlers that Cloudflare rates as Accountable (currently Apple's, Google's and Microsoft's) stay allowed for search.
• Every other training crawler is blocked, including the training-only crawlers run by Amazon, Anthropic, Meta and OpenAI. Those companies run separate search and training bots, so blocking the training bot doesn't touch search.

Cloudflare says "Apple, Google, and Microsoft honor or have committed (in a specified time frame) to honor this setting."

"Block" now reaches search crawlers too

This is the part to check. Block and Block on pages with ads now apply to mixed-use crawlers. In Cloudflare's words, selecting Block "will stop Applebot, Bingbot, and Googlebot from reaching your site — search included." If you want to stop training but stay findable, choose Disallow AI Training, not Block.

Do you need to do anything?

Cloudflare's answer is "Nothing, in almost every case. Your current settings carry over on their own." Specifically:

• If you used the old "Block AI Bots" switch (set to Block, or Block on pages with ads): search stays allowed, training becomes Disallow AI Training, and AI agents are blocked only on pages that show ads.
• If you'd already set the newer Training control to Block (or Block on pages with ads): "Previous Training selections of Block or Block on pages with ads will migrate to Disallow AI Training."
• If you never touched any of it: everything stays on Allow.

The old "Block AI Bots" switch is being deprecated in favour of separate Search, Training and Agent controls, and Managed Robots.txt is being replaced by Bot Preference Sync. Cloudflare says the controls are available on all plans.

Worth a look anyway: open your domain's Security Settings and confirm the Training control reads Disallow AI Training, not Block, unless you really do want out of search entirely. If you used the old switch to keep all AI bots out, also check the Agent control: after the migration, AI agents are only blocked on pages that show ads.

The Bing gap

One crawler doesn't fully work yet. Microsoft is building support for a robots.txt "no training" preference, targeted for early 2027. Until then, Cloudflare says, Disallow AI Training "will not automatically convey a no-training preference to Bing through robots.txt." For now, Cloudflare points site owners to Bing's NOARCHIVE meta tag, or Bing's Block URLs or Content Removal tool, to opt out of training. Cloudflare notes this is the same practical behaviour as before, because the previous Training Block setting didn't apply to Bingbot either.

What "Accountable" means

To get the label, a crawler operator must meet, or commit to meeting, four requirements:

• a way to opt out of AI training, through robots.txt or a similar standard;
• a way to opt out of AI summaries;
• URL-level visibility into which pages were made available for training, plus metrics on how content appeared in search;
• an assurance that opting out of training won't affect traditional search results.

Cloudflare says Apple, Google and Microsoft meet this through a mix of what exists today and time-bound commitments. Apple, for example, doesn't yet offer URL-level inspection but has described a solution in progress for next year. Cloudflare also rates the separate crawlers run by Amazon, Anthropic, Meta and OpenAI as Accountable. Cloudflare says it tracks the operators on its Radar service.

New sites get a preset

From 15 September, new domains are offered a preset depending on whether the site makes money from ads. Ad-supported sites get the stricter version: training set to Disallow AI Training, and AI agents blocked on pages that show ads. Cloudflare's reasoning is that ad revenue needs a human to actually see the page. You can change any of it.

What's next: AI summaries

Training is only half the question for publishers. The other is whether search engines summarise your content so readers never click through. Cloudflare says that by early next year it aims to let site owners control how much of their content is included in AI summaries, set once on Cloudflare rather than separately with each company.

Why it matters

In our view, for most site owners the forced choice between being found and being trained on was the real problem. This doesn't settle whether AI companies should pay for what they crawl. But it does turn "no training, yes search" into a setting, backed, Cloudflare says, by commitments from the three companies whose search traffic most sites can't do without. Check that your Block isn't quietly set to block search as well.