CPU Contention and Noisy Neighbors

The noisy neighbor effect is a problem in Linux. The key is to monitor how long a runnable thread waits for a physical core it cannot see.

Ruin is the destination toward which all men rush, each pursuing his own best interest in a society that believes in the freedom of the commons. - Garrett Hardin

Some time in the past, I worked at a company that self-hosted almost everything from whether it was a good decision or not, deserves a separate discussion.

Boxing Day morning, an incident happened. We had a huge backlog of messages in the RabbitMQ queue, and the cluster had lost quorum. Metrics looked right, yet nodes were failing heartbeats. Connected to the machine to diagnose, one glance at top told the real story. %st was consistently high, alternating values between 70-80%; we were experiencing the noisy neighbor effect.

What Contention Really ...