Anubis guards gates against hordes of LLM bot crawlers

Anubis is a sort of CAPTCHA test, but flipped: instead of checking visitors are human, it aims to make web crawling prohibitively expensive for companies trying to feed their hungry LLM bots.

It's is a clever response to a growing problem: the ever expanding list of companies who want to sell "AI" bots powered by Large Language Models (LLMs). LLMs are built from a "corpus," a very large database of human-written text. To keep updating the model, an LLM bot-herder needs fresh text for their "corpus."

Anubis is named after the ancient Egyptian jackal-headed god who weighed the heart of the dead, to determine their fitness. To protect websites from AI crawlers, the Anubis software weighs their willingness to do some computation, in what is called a proof of work challenge.

A human visitor merely sees a jackal-styled animé girl for a moment, while their browser solves a cryptographic ...