Google Cloud positions networking as the flexible glue for AI workloads, where accelerator availability is often the bottleneck. The blog explains how the ‘fluid compute’ idea lets you design an AI deployment around multiple compute options rather than a single accelerator type. It uses the example of running inference for a private LLM on NVIDIA B200 GPUs in A4 VMs (a4-highgpu-8g), and then details the resource-acquisition paths and the networking stacks that match each accelerator choice.

To obtain scarce capacity, Google Cloud offers several options: Dynamic Workload Scheduler Flex-start queues workloads until all required accelerator nodes are available at the same time, then provisions them together for non-preemptible runs of up to seven days. Dynamic Workload Scheduler calendar mode reserves capacity 1 to 90 days in advance with guaranteed start and end times, suited for scheduled pre-training and benchmarking. Future reservations guarantee access to committed hardware in a specified zone from a future date. Flex reservations provide short-term commitment windows without multi-year lock-in. In GKE, ComputeClasses allow multi-family fallback lists that automatically try alternative accelerator types if the primary pool is constrained regionally. Spot VMs offer discounted surplus compute for fault-tolerant, checkpointed batch jobs.

Networking is not the same across accelerators, and the article describes four configurations. Standard networking supports NVIDIA T4 (N1), L4 (G2), A100 (A2 single-node and multi-node), Cloud TPU v3, and single-host Cloud TPU v5e. Nodes communicate over the primary VPC using gVNIC over standard TCP/IP. This works well for distributed data preprocessing, decoupled pipeline stages, independent inference replicas, and computer vision workloads with standard VPC routing and network policies.

For distributed training and multi-node inference, accelerated GPU networking is required. GPUDirect-TCPX and TCPXO fabrics support NVIDIA H100 on A3 High VMs with 4 rails and H100 Mega on A3 Mega VMs with 8 rails. They use custom GPUDirect-TCPX/TCPXO offload engines over standard Ethernet, providing high-throughput multi-rail GPU communication without native RDMA hardware. These multi-VPC topologies can be deployed via pre-built blueprints from Cluster Toolkit. RoCEv2 fabrics support NVIDIA H200 (A3 Ultra), B200 (A4), GB200 NVL72 (A4X), and GB300 (A4X Max Bare Metal). They run over a dedicated RDMA VPC attached to a zonal network profile: “ZONE-vpc-roce” for VM instances and “ZONE-vpc-roce-metal” for bare metal. The VPC is isolated strictly for GPU communication, with rail-aligned subnets mapped to accelerator NICs, Jumbo Frames at MTU 8896, and non-blocking multi-terabit bandwidth. On GKE, the managed DRANET can automatically provision additional networks and drivers, mapping RDMA interfaces to GPUs for consumption via Kubernetes resource claims. Cluster Toolkit blueprints automate the full stack including RDMA VPCs, MTU tuning, and DRA drivers.

TPU networking differs from GPU networking. Supported accelerators include Cloud TPU v4, v5p, v5e multi-host Pod slices, v6e (Trillium), and TPU7x (Ironwood). Inside a TPU Pod or slice, chips communicate directly over dedicated ultra-low-latency optical links arranged in 2D or 3D torus meshes, bypassing traditional network stacks. In TPU v4 and v5p SuperPods, optical circuit switches (OCS) dynamically change physical topologies, route around faulty trays, and provision custom-sized slices without manual recabling. Starting with TPU v6e, a native multi-NIC architecture isolates Kubernetes management traffic on the primary VPC while secondary dedicated VPCs handle high-throughput TPU data and cross-slice communication. DRANET is also available for TPU deployments on GKE. For models larger than a single slice, Cloud TPU Multislice connects independent ICI meshes over Google’s Jupiter Data Center Network using these dedicated multi-NIC paths.

Finally, Cloud Run GPU services support NVIDIA L4 (G2) and NVIDIA RTX PRO 6000 (Blackwell). Direct VPC Egress binds serverless containers to a private VPC using sub-minute IP allocation, enabling secure, low-latency access to internal data lakes, databases, and private APIs without traversing the public internet or using legacy connector VMs.