Cloudflare Gives Website Owners Option to Charge OpenAI Bots for Scraping

Cloudflare has launched a private beta feature called Pay per Crawl, whose sole purpose is to let a website owner charge an AI crawler a fixed fee each time the crawler requests a page. The feature addresses a common operational gap: currently, a publisher can either leave all content open to automated collection or block crawlers entirely, and any paid arrangement must be negotiated manually. 

At Belitsoft, a software development company for media firms and expert in Cloudflare integrations, we have already explored the legal challenges facing AI companies. See my previous article, OpenAI Data Retention Court Order: Implications for Everybody.

Pay per Crawl replaces that binary choice with an automated, per-request billing step handled entirely at Cloudflare’s edge. The mechanism revives HTTP status code 402, “Payment Required,” an unused element of the original web specification, and uses it to signal that a charge ...