Demand for AI infrastructure is outstripping supply, and accelerator shortages mean compute is often scattered across data centers, sometimes on different continents. At the same time, long-running agentic workloads with 100k-800k+ token context windows consume accelerator memory faster than ever, making it critical to maximize ‘intelligence per dollar.’ To address this, Google built a layered routing architecture that makes globally scattered capacity behave like a single pool. At the edge, the multi-cluster GKE Inference Gateway handles global traffic distribution and high availability. Beneath it, the LLM-d router runs memory-aware scheduling algorithms. The architecture is runtime-, model-, and accelerator-agnostic, working across serving frameworks, model families, and GPU or TPU hardware.

To validate the approach, they benchmarked a multi-region GKE deployment of 17,000 compute nodes across the US and Europe, serving a leading Mixture of Experts (MoE) model with SGLang. Scaling from one cluster to three clusters across regions achieved a near-linear throughput boost—from 0.72 req/s (2,898 tok/s) to 2.10 req/s (8,457 tok/s)—while maintaining a 99.9% success rate under heavy multi-client concurrency. Routing traffic through the Gateway added less than 1% overhead, delivering 99.5% of the throughput of a direct local cluster call.

The deployment spans three GKE clusters in us-east5 (config cluster), us-west8, and europe-west4, but clients see only a single global virtual IP. The Gateway routes traffic based on live application signals: the Endpoint Picker Proxy (EPP) reads KV-cache token utilization from the inference engines and emits it as a metric for the load balancer. When a region runs hot, traffic spills to the next healthy region. Distributed LLM engines use master-worker topologies where only rank-0 pods serve the API; GKE handles local routing with Services and LeaderWorkerSet (LWS), and the Gateway respects these topologies automatically.

Memory-aware routing is essential because round-robin load balancing treats every request equally, which is inadequate for LLMs. Heavy prompts saturate compute cores, long generations stress memory bandwidth, and long-context conversations exhaust VRAM. In the benchmark, as the primary region crossed a 40% KV-cache utilization threshold, the Gateway automatically routed overflow to healthier regions—no manual intervention. This pooling of scattered capacity translates directly into maximizing ‘intelligence per dollar,’ getting near-perfect proportional performance from each accelerator.

Five lessons stand out for teams planning distributed inference: smarter load balancing pays for itself; agentic workloads change the bottleneck from compute to memory; KV-cache utilization should be a first-class routing signal; routing layers must integrate with native serving patterns like LWS; and large-scale model builders should bet on an open, portable stack like LLM-d on GKE to absorb capacity wherever it lands.