Multi-Node KV Cache Offloading with GKE and Managed Lustre

Enterprise LLM serving faces a scaling wall when KV caches for long-context or agentic workloads exceed local CPU RAM and host SSD tiers. Pooling node-local SSDs into a distributed layer adds capacity but forces the compute cluster to manage complex data distribution and cross-node replication, introducing networking overhead and operational burden. The core tension is between keeping inference fast and keeping infrastructure manageable at scale.

Google Cloud presents a concrete alternative: offload the attention state to Managed Lustre, a dedicated high-performance parallel filesystem, using the llm-d offloading stack as a cluster-wide decentralized cache. On a six-node A3 Mega cluster running Llama-3.3-70B with 50K-token prompts, this approach delivers over 50% TCO savings and reduces GPU-hour requirements by nearly 60%, driven by a 95% cache hit rate. A hybrid tier that adds CPU RAM offload further improves Time to First Token (TTFT) by ~40% and cuts end-to-end latency by 30%. The post includes a complete deployment guide for GKE, with validated configurations for Qwen3.5-35B-A3B and Gemma 4, plus a PVC Evictor service that performs LRU garbage collection on the shared filesystem.

Serious builders should see this as a production-ready blueprint for decoupling KV cache scaling from node-level memory constraints. The architecture replaces complex pooled-SSD management with a proven parallel filesystem, making it practical to serve long-context agentic models without skyrocketing GPU costs. The explicit configuration details, performance numbers, and evictor sharding guidance (one replica per 72 TB of Lustre capacity) give engineering teams a concrete starting point for their own multi-node deployments.