How we cut Vertex AI latency by 35% with GKE Inference Gateway

As generative AI moves from experimentation to production, platform engineers face a universal challenge for inference serving: you need low latency, high throughput, and manageable costs. 

It is a difficult balance. Traffic patterns vary wildly, from complex coding tasks that require processing huge amounts of data, to quick, chatty conversations that demand instant replies. Standard infrastructure often struggles to handle both efficiently.

Our solution: To solve this, the Vertex AI engineering team adopted the GKE Inference Gateway. Built on the standard Kubernetes Gateway API, Inference Gateway solves the scale problem by adding two critical layers of intelligence:

1. Load-aware routing: It scrapes real-time metrics (like KV Cache utilization) directly from the model server's Prometheus endpoints to route requests to the pod that can serve them fastest.

2. Content-aware routing: It inspects request prefixes and routes to the pod that already has that context in its KV cache, avoiding expensive re-computation.

By ...