Cloudflare’s pay-per-crawl is built to fail. Here’s why

When Cloudflare announced its new Pay-Per-Crawl marketplace, some people saw a breakthrough. The idea is that if AI companies want to crawl your website to train their models, they should compensate you for the use of your content. As the CEO of a legal AI company recently sued for scraping public data, I’d love for this to work.

However, it won’t, at least not in this way.

Last year, my company, Caseway, was sued by CanLII, the operator of Canada’s free legal decisions database, for allegedly using publicly available court data without a license. I’ve had a front-row seat to the vagueness of the legal rules surrounding AI scraping. And I’ve watched the wave of litigation since. The New York Times sued OpenAI and Microsoft for using millions of their paywalled articles to train GPT-4.