This post describes two Google Cloud reference architectures for networking AI inference model serving: one for GKE-only backends and one for all other backend types. The goal is to simplify how multiple models are called while retaining centralized governance. Both designs use a common entry point: a stable, secure endpoint that acts as a control zone where policy, security, and logic can be enforced. It terminates TLS, integrates with API management, supports service extensions, and can use Model Armor for added security. The entry point can be either a Cloud Load Balancer or an Inference Gateway.

Common services in both designs include a Private Service Connect inference endpoint, which anchors the entry point inside the consumer VPC and keeps traffic on a private internal IP; optional Apigee API Management, integrated via an Apigee Extension Processor callout to verify client identity, rate limits, and quotas before requests reach compute resources; and Model Armor, which screens prompts and completions against prompt injection and sensitive data leakage.

For GKE-only deployments, the design centers on the GKE Inference Gateway, deployed as an internal Application Load Balancer (gke-l7-rilb). The gateway acts as a specialized ingress engine: it parses the incoming request payload, evaluates HTTPRoute rules, and routes queries to the appropriate inference pool. An inference pool is a logical group of replicas of the same model, with an initial size and optional autoscaling. Individual replicas run on single- or multi-node GPU/TPU node pools, and a replica set is a uniform group of these instances.

In the GKE traffic flow, a client in the consumer VPC makes an OpenAI-compatible API call to the Private Service Connect endpoint, routing directly to the gateway through a regional internal Application Load Balancer. The gateway reads the target model parameter in the request body and adds it to HTTP headers. If Apigee is present, it checks credentials and quotas; Model Armor screens the prompt. The gateway evaluates HTTPRoute mappings to find the target pool, matches shared prefix cache context, and routes to the lowest-load GPU/TPU replica based on real-time Prometheus data. The replica runs inference, output tokens pass through Model Armor for final response verification, and the response streams back over the private connection.

The all-backends design targets mixed environments: GKE, Cloud Run, Agent Platform, on-premises data centers, or external clouds. A regional internal Application Load Balancer acts as the central Layer 7 routing proxy, handling routing logic, SSL termination, and Service Extensions callouts. Because an Application Load Balancer needs external help for body-based routing, an Inference Payload Processor implemented as a Cloud Run callout inspects the JSON body of OpenAI API requests, extracts the target model identifier, and writes an X-Gateway-Model-Name header that drives URL map routing. A Network Endpoint Group (NEG) then delivers flexible routing to heterogeneous backends based on that injected header.

In the all-backends flow, the client targets the Private Service Connect endpoint, the regional internal ALB receives the request, and sends the payload to the Cloud Run callout, which injects the model header. The request passes to Apigee for identity and quota validation, then to Model Armor for data scrubbing and malicious prompt blocking. The load balancer’s URL map inspects the model header and forwards the request to the matching backend NEG (Agent Platform, GKE, Cloud Run, Hybrid, or Internet). The backend executes the prompt, Model Armor screens the completion, and the result returns privately along the ingress path.

Both architectures keep inference traffic inside the private network, apply consistent policy and safety checks, and differ mainly in where the routing intelligence lives: the GKE Inference Gateway for GKE-only, or an ALB plus a Cloud Run callout and NEGs for heterogeneous backends.