Cloudflare experiment will block AI bot scrapers unless they pay a fee

Cloudflare has rolled out a couple of new measures meant to keep AI bot crawlers at bay. To start with, every new domain customer that signs up with the company to manage their website traffic will now be asked if they want to allow AI crawlers or to block them altogether. The company released a free tool in 2024 to block AI bots, but with this change, users can block them by default without having to tinker with their settings. Several big publishers, including Condé Nast, TIME and The Associated Press have already signed up to block crawlers. In addition, Cloudflare has launched a private beta experiment called "pay per crawl," which would only allow crawlers to access a website's content if they pay for it.

Matthew Prince, Cloudflare's CEO, recently went on record to say that publishers are facing an existential threat ...