Content Creators, Want to Charge or Block AI Crawlers? Here’s Cloudflare’s Solution

Cloudflare’s new pay per crawl system, which also applies to site owners, is currently in a private beta.

Cloudflare has introduced a new system that aims to give content creators and website owners more control over the AI crawlers that consume their web content daily. Currently in a private beta, the proposed Cloudflare’s pay-per-crawl system would support the ongoing development of next-gen AI platforms while ensuring fair compensation across the board.

Analyzing the pay-per-crawl model

Content creators and website owners have been struggling to come to terms with AI crawlers since their inception. While some choose to make their content freely available for AI training and reference, others opt to block AI crawlers entirely. However, Cloudflare’s new model gives content creators and website owners the option of charging a fee for access.

Cloudflare’s blog post launching pay-per-crawl published today reads, in part: “We ...