Cloudflare, a web infrastructure and security company, has announced that AI crawlers will now be blocked by default across websites using its platform, shifting from an opt-in to a permission-based model. This change eliminates the need for content owners to manually adjust settings to prevent unauthorized data scraping. “Our goal is to put the power back in the hands of creators, while still helping AI companies innovate,” said Matthew Prince, Co-Founder and CEO of Cloudflare. “This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone.” Prince emphasized the need for a sustainable economic framework that supports content creators as AI continues to reshape the digital landscape.
In addition to default blocking, Cloudflare is testing a pay-per-crawl system in private beta, offering content owners the ability to monetize content accessed by AI crawlers. Crawlers will be met with payment prompts or receive a 402 HTTP error if access is denied. Publishers can choose to block, allow, or charge for crawler access, with flexibility for negotiated content partnerships. Cloudflare stated in a blog post, “Pay per crawl begins a technical shift in how content is controlled online.” The company anticipates varied pricing models and digital marketplaces to emerge as the model matures.
