Cloudflare broke itself – and a big chunk of the Internet – with a bad database query

Cloudflare CEO Matthew Prince has admitted that the cause of its massive Tuesday outage was a change to database permissions, and that the company initially thought the symptoms of that adjustment indicated it was the target of a “hyper-scale DDoS attack,” before figuring out the real problem.

Prince has penned a late Tuesday post that explains the incident was “triggered by a change to one of our database systems' permissions which caused the database to output multiple entries into a ‘feature file’ used by our Bot Management system.”

The file describes malicious bot activity and Cloudflare distributes it so the software that runs its routing infrastructure is aware of emerging threats.

Changing database permissions caused the size of the feature file to double and grow beyond the file size limit Cloudflare imposes on its software. When that code saw the illegally large feature file, it failed.

And then it recovered ...