The world's largest so-called content delivery network, American Cloudflare, has set a deadline that is fast approaching.
On Tuesday, September 15, 2026, Cloudflare will by default start blocking all bots that use the data they collect for training or developing AI services.
Cloudflare itself may not mean anything to the average internet user, but about a third of all significant websites in the world use Cloudflare as a kind of doorman for their content, which lightens the load on the sites' own servers - and blocks unwanted bot traffic.
And starting September 15, bots that collect data for "mixed use" purposes will also be included in that group of "unwanted bot traffic".
This category includes advertising giant Google's own bot, Googlebot.
Google uses its data-collecting bot to create the index for its traditional search engine, but the same bot also collects data for Google's AI search.
As Cloudflare's rules change on September 15, Google's bots that collect and index website content will therefore be blocked by default, unless the site administrator has decided to specifically allow them. Cloudflare calls the upcoming change day Content Independence Day.
It is assumed that a fairly large portion of website administrators will do nothing about the default settings - and some sites may block Google's bot in the future for principled reasons. This means they will practically disappear from Google - and also from Bing, among others - immediately.
The background, of course, is the fact that after Google shifted from its traditional link-based search results to AI-generated summaries, which kind of "broke the unspoken contract" between websites and Google.
Previously, websites gladly provided their content for Google's search engine, because Google reciprocally sent vast numbers of users from its search results back to the websites. And because the internet largely operates on advertising revenue, the arrangement was profitable for both sides: virtually all websites could be found through Google's search engine, and websites gained more readers via Google.
But in the new world of AI summaries, a large portion of users already get the information they are looking for directly from Google's AI summary - meaning they have no reason to click through to the source of the information. Thus, websites whose information forms the basis of the answers receive no readers from Google - and consequently no advertising revenue either.
Cloudflare has strongly sided with website publishers in this matter and aims to, for its part, force AI companies to pay for the content they scrape from websites for use in their AI models. The upcoming change to default settings is therefore the first serious challenge to Google and other AI companies, through which they will be pressured to pay for the data they collect from the web for their use.
Cloudflare's setting, which blocks "mixed-use" data collection, is already available to websites, but now that it's becoming the default, it could indeed happen that millions of websites disappear from Google in one fell swoop after September 15.
Cloudflare itself may not mean anything to the average internet user, but about a third of all significant websites in the world use Cloudflare as a kind of doorman for their content, which lightens the load on the sites' own servers - and blocks unwanted bot traffic.
And starting September 15, bots that collect data for "mixed use" purposes will also be included in that group of "unwanted bot traffic".
This category includes advertising giant Google's own bot, Googlebot.
Google uses its data-collecting bot to create the index for its traditional search engine, but the same bot also collects data for Google's AI search.
As Cloudflare's rules change on September 15, Google's bots that collect and index website content will therefore be blocked by default, unless the site administrator has decided to specifically allow them. Cloudflare calls the upcoming change day Content Independence Day.
It is assumed that a fairly large portion of website administrators will do nothing about the default settings - and some sites may block Google's bot in the future for principled reasons. This means they will practically disappear from Google - and also from Bing, among others - immediately.
The background, of course, is the fact that after Google shifted from its traditional link-based search results to AI-generated summaries, which kind of "broke the unspoken contract" between websites and Google.
Previously, websites gladly provided their content for Google's search engine, because Google reciprocally sent vast numbers of users from its search results back to the websites. And because the internet largely operates on advertising revenue, the arrangement was profitable for both sides: virtually all websites could be found through Google's search engine, and websites gained more readers via Google.
But in the new world of AI summaries, a large portion of users already get the information they are looking for directly from Google's AI summary - meaning they have no reason to click through to the source of the information. Thus, websites whose information forms the basis of the answers receive no readers from Google - and consequently no advertising revenue either.
Cloudflare has strongly sided with website publishers in this matter and aims to, for its part, force AI companies to pay for the content they scrape from websites for use in their AI models. The upcoming change to default settings is therefore the first serious challenge to Google and other AI companies, through which they will be pressured to pay for the data they collect from the web for their use.
Cloudflare's setting, which blocks "mixed-use" data collection, is already available to websites, but now that it's becoming the default, it could indeed happen that millions of websites disappear from Google in one fell swoop after September 15.
