Key facts
- Cloudflare will block AI crawlers from accessing ad-supported content by default starting September 15, 2026.
- The policy targets "mixed-use" crawlers that blend search, AI agent use, and training.
- Website owners can opt-out of this default blocking by adjusting Cloudflare settings.
- The change aims to enable publishers to be compensated for their content used by AI companies.
- Cloudflare is partnering with Ceramic.ai and You.com to implement payment mechanisms for publishers.
Cloudflare has announced a new policy that will block AI crawlers from accessing ad-supported web content by default starting September 15, 2026. The company stated that this measure is intended to create a sustainable ecosystem where publishers can be compensated for their content used by AI models and agents.
Under the new default settings, "mixed-use" crawlers that combine traditional search functions with AI agent use and training will be prevented from crawling pages that host advertisements, unless website owners manually adjust their settings. This policy will affect new Cloudflare customers, new sites set up by existing customers, and all current free customers.
Cloudflare CEO Matthew Prince highlighted that the majority of internet traffic is now non-human and emphasized the need for faster action to foster a sustainable ecosystem. He noted that the new tools and partnerships aim to provide website owners with greater visibility and commercial opportunities, while benefiting AI companies with clear intentions. Cloudflare hopes this will encourage mixed-use crawlers to separate search functions from agent use and training.
The company pointed out that many website owners wish for their content to be discoverable through both search engines and AI services but want to prevent their intellectual property from being used without compensation. Cloudflare specifically referenced Google, suggesting it has an advantage due to its difficulty in allowing discoverability without content being used for AI training.
Google has previously stated that its Google Extended bot allows site owners to opt out of content usage for AI training without impacting search inclusion. However, its primary Googlebot still crawls for Search, including AI features. Cloudflare also noted that over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages, suggesting the new policy could also conserve bandwidth and compute resources.
To implement this, Cloudflare is collaborating with Ceramic.ai and You.com. Publishers who opt in will be paid when their content appears in Ceramic's AI search results or when You.com accesses their premium content. Other AI companies can customize similar models for their operations.