Open AI-native Kubernetes stack for RL, inference, and agents

Over the past decade, Kubernetes established itself as the standard for orchestrating microservices and containerized applications. The post argues that AI workloads are now moving from isolated research prototypes into large-scale, multi-tenant production fleets, and presents Google Cloud’s open-source, Kubernetes-native stack for this shift, to be demonstrated at KubeCon + CloudNativeCon North America 2026 in Salt Lake City.

The motivation is production AI moving beyond closed black-box APIs. Teams want to run open-weight models such as the Gemma family, adapt them with reinforcement learning on proprietary datasets, and serve them with predictable latency and high throughput, using examples like DolphinGemma for dolphin communication research, legal platforms evaluating contracts, and autonomous coding agents. The main hurdles are accelerator underutilization in RL when alternating training and rollout generation leaves GPUs/TPUs idle; cache-blind load balancing in standard Kubernetes routing that destroys KV cache locality for multi-turn agent calls; framework fragmentation from parallel Ray/Slurm clusters; and agent security when executing untrusted model-generated shell commands in shared clusters.

To address these, Google Cloud is introducing and expanding a unified open-source stack: OpenRL (CNCF Sandbox) is a Kubernetes-native post-training service that simplifies RL post-training. It exposes a clean four-primitive Python abstraction (forward_backward, optim_step, sample, save_state) so researchers write standard PyTorch code without managing Kubernetes internals. It uses cooperative accelerator time-slicing to interleave training steps and rollout inference on the same physical accelerators, avoiding separate static clusters. Delta-weight synchronization ships only parameter deltas to rollout workers, eliminating rollout cold starts. Zero-config dynamic resource allocation profiles context lengths, batch dimensions, and LoRA/full fine-tuning configurations, then bin-packs workloads across heterogeneous accelerators using Kubernetes DRA claims.

llm-d 1.0 (CNCF Sandbox) is an open distributed inference control plane co-founded by Google Cloud, Red Hat, IBM Research, CoreWeave, and NVIDIA, with contributions from over 500 engineers across 25+ organizations. It transforms standalone inference engines like vLLM and SGLang into a coordinated cluster-wide serving fabric. Its Endpoint Picker integrates into the Kubernetes Gateway API Inference Extension and is managed natively through GKE Inference Gateway. In production, it doubled KV cache hit rates from 35% to 70% and cut median time-to-first-token by 70-85% on multi-turn agent workloads. It also provides 3-tier KV cache offload (HBM to host CPU RAM to local NVMe SSD / Google Cloud Managed Lustre / Cloud Storage), EPD Disaggregation with DisaggregatedSet built on LeaderWorkerSet, native TorchTPU upstream support for PyTorch on TPU v5e, v6e, v7x alongside NVIDIA GPUs, latency-tolerant async processing via llm-d-async-processor with traffic prioritization, and fleet observability through Prism for throughput, inter-token latency, TTFT, and cache hit ratios.

KubeRay is a Kubernetes-native operator managing the full lifecycle of Ray clusters, jobs, and services via RayCluster, RayJob, and RayService custom resources. It provides declarative cluster lifecycle, integration with Kueue for multi-tenant fair sharing, preemption, and quotas, and elastic autoscaling based on active Ray tasks and actor placement.

For agent security, Agent Substrate (CNCF Sandbox) and gVisor provide kernel-level sandboxing. gVisor is a user-space kernel that intercepts and virtualizes system calls, preventing untrusted agent code from compromising the host kernel or neighboring tenants. Sandboxes spin up in milliseconds per agent turn, supporting both RL rollout generation in OpenRL and live production agent workflows.

The post reports a benchmark evaluation using Gemma 4 against the Legal Agent Benchmark (LAB) developed by Harvey.ai, covering 24 legal practice areas from M&A due diligence to contract dispute analysis. Standard SFT plateaued around 50% accuracy on multi-step reasoning rubrics, while reproducible RL loops with OpenRL lifted accuracy to 70%. The training pipeline dynamically shared GPU and TPU capacity with concurrent LoRA and inference jobs on GKE without manual partition management.

Adoption patterns are summarized for four organization personas: enterprise platform teams can turn existing GKE clusters into self-hosted Reinforcement Learning-as-a-Service using DRA, Kueue, and accelerator time-slicing; regulated enterprises can post-train and serve 1B-30B Gemma models inside a private VPC with domain-specific reward functions; AI-native builders can deploy llm-d prefix-cache routing paired with gVisor-isolated Agent Sandbox execution targets; and model-as-a-service/neocloud providers can expose OpenRL multi-tenant APIs and llm-d‘s multi-accelerator control plane across mixed GPU and TPU clusters.

The article closes with KubeCon North America 2026 booth and session information, plus getting-started links for OpenRL, llm-d, KubeRay, and Gemma on GKE.

The open AI-native stack on Kubernetes: From open models to distributed inference and agents

View Original