Cloudflare creates AI crawler tollbooth to pay publishers

ai-pocalypse Cloudflare has started blocking AI web crawlers by default in a bid to become the internet's gatekeeper.

The term "gatekeeper" has been applied in a pejorative sense to platform companies like Apple and Google that use their contractual and technical control over operating systems to extract monopoly rents from developers within the platform ecosystem.

Cloudflare proposes an alternative interpretation of that role that's less self-serving: to help protect online publishers from the predation of AI firms.

CEO Matthew Prince in a blog post explains that the deal Google made with content creators almost 30 years ago was that the search service would send traffic to websites in exchange for the opportunity to access and index their content.

That deal, he argues, has been broken as Google and rival AI companies deploy search services derived from uncompensated web content crawls, services that starve publishers of revenue by referring ...