At the PyTorch Conference, Google and partners released llm-d 1.0, an open-source, Kubernetes-native stack for distributed LLM inference. The CNCF Sandbox project, co-founded by Google with Red Hat and IBM, draws on a community of more than 500 contributors across 25 organizations and connects PyTorch-based serving engines like vLLM and SGLang with Kubernetes orchestration. llm-d 1.0 is aimed at the shift from single-turn chat to multi-step agents and reinforcement learning rollouts, which fan out across tools, share long prefixes, and arrive in bursts; serving them on a single machine or behind a generic round-robin load balancer leads to duplicated compute and memory pressure.

llm-d turns a cluster of model servers into a coordinated serving pool. The 1.0 release provides a hardened request path, v1 APIs, prefix- and load-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism, and batch inference across Google Cloud TPUs and GPUs. It also adds router high availability, zero-downtime updates, and DisaggregatedSet for prefill/decode-aware rolling updates that preserve context. Agent-aware serving includes session-aware routing, turn-priority scheduling, and native Responses API state, while TorchTPU lets upstream vLLM and SGLang run PyTorch models natively on TPUs without model rewrites.

The routing layer builds on Gateway API Inference. On Google Kubernetes Engine, GKE Inference Gateway runs the llm-d router as a managed service, extending routing across regions. The router tracks which replicas hold each cached prefix and weighs load and predicted latency when picking a replica. When accelerator memory fills, llm-d uses vLLM’s OffloadingConnector, Mooncake, or SGLang’s HiCache to move KV cache to host memory and then to SSD or Google Cloud Managed Lustre. The post reports that on an agentic coding workload on Ironwood TPU (TPU 7x), prefix-aware routing plus host memory offload cut median time-to-first-token by more than 85% and raised total throughput 2.4x versus a plain Kubernetes Service, as of October 2026.

Prefill, decode, and multimodal encode run as separate pools, managed by DisaggregatedSet on top of LeaderWorkerSet. For latency-tolerant traffic, llm-d 1.0 introduces llm-d-async-processor, which pulls requests from queues like Pub/Sub or Redis and uses queue and pool-dispatch gates to enforce priority lanes so background jobs run without starving interactive users. Time-slicing for TPUs and GPUs interleaves async inference into idle cycles during RL rollout generation or weight updates, raising accelerator duty cycles from about 40% to over 70%.

The post includes several production results. Gemini platform (formerly Vertex AI) serves production traffic through GKE Inference Gateway, doubling prefix cache hit rate from 35% to 70%, cutting TTFT by more than 35% for coding agents, and cutting P95 TTFT by more than 50% under bursty chat traffic. In a 17,000-node multi-region GKE deployment serving a mixture-of-experts model in the US and Europe, multi-cluster GKE Inference Gateway and the llm-d router scaled throughput 2.9x across three clusters with a 99.9% success rate and less than 1% routing overhead. On an A3 Mega cluster serving a 70B-parameter model, multi-connector KV cache offloading to Google Cloud Managed Lustre plus host CPU RAM delivered about a 40% improvement in TTFT and a 30% reduction in end-to-end latency on 50k-token prompts compared with CPU RAM offload only, while reducing GPU-hour requirements by nearly 60%.

Adopters report similar gains. Snap says it is integrating llm-d into production AI infrastructure and achieving prefix cache hit rates up to 75-80% with Envoy-based service mesh integration. ZenteiQ reports over 4x higher aggregate inference throughput on GKE Inference Gateway, increasing daily generated token capacity from 1.47 billion to 5.97 billion tokens per 1,000 TPU v6e accelerators. Mistral says it uses llm-d‘s prefix-cache-aware routing, flow control, and disaggregated serving to enforce priority bands across commercial tiers and plans to contribute to llm-d and DisaggregatedSet. The components are composable, so teams can keep their engine, model, and cluster and adopt only the serving pieces they need. Prism, llm-d‘s open-source benchmark dashboard, provides standardized recipes and lets users compare runs on their own clusters. The post closes by directing readers to deploy llm-d 1.0 on GKE with verified Helm charts and to start with prefix-aware routing in front of an existing vLLM deployment.