Cloudflare has set a new deadline for the AI sector to differentiate between web crawlers utilized for traditional search functions, such as Google Search, and those employed for AI agents and training. Beginning on September 15, 2026, Cloudflare’s standard configurations will prevent “mixed-use” crawlers from accessing any pages that display advertisements, the company revealed on Wednesday.

This signifies that crawlers that integrate search, agent usage, and training will be barred from crawling these websites by default unless the website owner modifies the settings accordingly. These default changes will impact new Cloudflare clientele, new sites established by current users, and all pre-existing free customers, according to the company’s statement.

This initiative could influence how AI model developers access web material for training and support their agent-based services.

Cloudflare emphasizes that the majority of website proprietors desire their material to be easily found through search and often through AI services as well; however, they seek protection against their intellectual property being distributed freely.

Cloudflare explicitly mentions the “largest search engine in the world” (clearly a reference to Google!) as having access to roughly “twice the amount of information” compared to other AI firms because the search titan complicates discoverability for clients without incorporating AI usage.

Google has previously contested this assertion, highlighting that it offers a bot named Google Extended, which allows site owners to opt out of having their content utilized for training and AI applications like Gemini Apps and Vertex API. Utilizing it doesn’t affect a site’s presence in Google Search. Nonetheless, the tech behemoth’s primary Googlebot continues to crawl for Search, including AI capabilities such as AI Overviews and AI Mode.

“Given that the overwhelming majority of online traffic is now from non-human sources, we must accelerate our efforts so that a sustainable ecosystem can develop,” stated Cloudflare co-founder and CEO Matthew Prince in his announcement regarding the recent development, referencing the noteworthy moment when bot traffic exceeded human traffic online for the first time. This transition was not anticipated until the following year.

“Cloudflare’s latest tools and collaborations provide website owners with enhanced visibility and commercial opportunities, benefiting AI companies that possess bots with clear and transparent purposes. We anticipate that the proposed default modifications will prompt mixed-use crawlers to differentiate between search and agent usage as well as training,” Prince remarked.

While Cloudflare provides numerous products to assist users in launching their own AI systems, the company has also unveiled a variety of tools to give publishers more authority over their content in the AI landscape. In recent years, Cloudflare has introduced tools aimed at combating AI bots, including a marketplace that enables websites to charge AI bots for scraping, known as Pay Per Crawl.

This model is now evolving into “Pay Per Use,” the company disclosed, allowing publishers to charge AI firms when their content generates value, not merely when it is retrieved.

This alteration could also aid in conserving publishers’ bandwidth and computing resources for AI model providers, as Cloudflare’s data indicated that over 50% of crawling traffic from AI crawlers is utilized in re-fetching unchanged pages.

To execute this initiative, Cloudflare is initially collaborating with two partners, Ceramic.ai and You.com. When a publisher opts in, they receive compensation when their content is featured in Ceramic’s AI search results or when You.com retrieves a piece of their premium content.

Other AI organizations may tailor this model to fit their operational methods, claims Cloudflare.