A Guide to AI Cold Starts on Cloud Run

Developers on Reddit reported AI cold starts on Cloud Run taking up to 20 seconds, a frustrating gap that made some abandon serverless GPUs for GKE. The article breaks this down into four phases: infrastructure provisioning (~5s), container image streaming (1-2s), engine initialization (5-15s), and model loading into VRAM. The real tension is between the convenience of scale-to-zero and the latency of moving gigabytes of model weights, which is fundamentally different from standard web microservices.

To mitigate this, the article provides concrete tactics for each phase. 4-bit quantization and fast formats like GGUF and Safetensors directly shorten Phase 4 by reducing weight sizes. Startup CPU boost doubles CPU during startup to accelerate engine initialization (Phase 3), while Direct VPC Egress and Private Google Access keep weight traffic on Google’s backbone. Concurrency tuning uses a formula from Google to avoid unnecessary scale-outs: (model instances parallel queries) + (model instances batch size). Elastic‘s production strategies include setting enforce_eager=True in vLLM to trade throughput for sub-minute cold starts, pre-merging LoRA checkpoints, and deploying each workload as its own Cloud Run service.

The takeaway is that treating GPUs as fungible compute rather than infrastructure to manage is key. With the right optimizations—quantization, startup CPU boost, and concurrency tuning—cold starts become a manageable tradeoff rather than a blocker. Cloud Run handles NVIDIA drivers and CUDA, starting instances in about 5 seconds, and features like a 15-minute grace period and proactive “wake-up calls” via non-inference endpoints can further mask latency. Serious builders should view these patterns as a blueprint for production-grade serverless AI inference.