Perplexity's Bots Ignore No-Crawl Rules, Says Cloudflare

Allegations of Improper Data Collection Are Not New for Perplexity Rashmi Ramesh (rashmiramesh_) • August 5, 2025

Artificial intelligence firm Perplexity is facing allegations it sidestepped long-standing internet norms in pursuit of data. Cloudflare accused the AI search engine startup of positioning itself as a Google alternative, ignoring website restrictions and disguising its scraping activities.

See Also: Post-Quantum Cryptography - A Fundamental Pillar in the Future of Cybersecurity [ES]

The network security and infrastructure company's engineers on Monday outlined what they described as behavior consistent with attempts to bypass content restrictions. Despite publishers explicitly disallowing Perplexity in their robots.txt files - the decades-old standard used to signal what content should not be indexed or scraped - the company allegedly continued to access site content.

"This undeclared crawler utilized multiple IPs not listed in Perplexity's official IP range," Cloudflare wrote. Researchers said Perplexity's bots rotated IP addresses and altered ...