Partitioning Large Messages and Normalizing Workloads Can Boost Your AWS CloudWatch Ingestion

Sriram Madapusi Vasudevan works as a senior engineer at AWS. He developed a new way to deal with latency spikes in large data ingestion systems. He found that large, low-priority messages were crowding out smaller, higher-priority ones.

In large-scale data ingestion systems, small architecture choices can have dramatic performance implications.

During my time at AWS CloudWatch, we were in the midst of a migration from our legacy metric stack to a spanky new one. I was the on call engineer as our alarms blared: end-to-end latency spikes had breached a critical threshold. A quick partitioning tweak later, those noise-making spikes vanished and throughput climbed 30% on the same hardware. In this deep-dive, you’ll see exactly how I diagnosed a flawed “uniform message” assumption and turned it into high-volume reliability.

The System Architecture

The data pipeline processed messages from a number of queues ...