Improving Ray Serve LLM on GKE: 5x Throughput, 8x Latency

The long-standing tension between Ray Serve‘s Python-native developer experience and its inference performance has been a practical headache for teams deploying LLMs at scale on Kubernetes. Users often had to choose between Ray’s flexible orchestration or the raw throughput of a bare-metal vLLM setup. Google Kubernetes Engine and Anyscale directly address this tradeoff with architectural changes that eliminate proxy overhead and streaming bottlenecks.

Three concrete optimizations underpin the improvements: HAProxy integration for internal load balancing reduces Python runtime saturation; a direct token streaming architecture bypasses the ingress router for return tokens, slashing latency; and the v2 Ray executor for vLLM moves Ray out of the data plane, enabling asynchronous scheduling. Benchmarks on GKE using Gemma 4 E2B on A4 VMs (NVIDIA HGX B200) show up to 5x higher throughput and 8x lower latency compared to previous Ray Serve, approaching native vLLM performance.

The practical takeaway is that developers no longer need to compromise between ease of use and performance for LLM serving. These optimizations make Ray Serve on GKE a credible choice for production inference, combining Ray’s distributed programming model with latency and throughput competitive with hand-tuned vLLM deployments. Teams evaluating inference infrastructure should revisit Ray Serve if they previously dismissed it due to overhead.