Modal CTO on the 100,000 Sandbox Problem and AI Infrastructure

Traditional cloud infrastructure, and particularly Kubernetes, was never designed for the bursty, compute-heavy workloads that define modern AI. Modal CTO Akshat Bubna explains that the paradigm of long-running services and predictable resource usage simply breaks down when you need to spin up 100,000 sandboxes for an RL rollout or handle the erratic demand of custom inference. This mismatch between cloud assumptions and AI reality is the core tension Modal set out to solve, starting long before ChatGPT popularized the need for elastic GPU compute.

Modal’s answer is a purpose-built AI cloud that replaces Kubernetes with serverless functions, decorator-based infrastructure, and a set of primitives specifically for AI workloads. They offer GPU snapshotting to nearly eliminate cold starts, DeFlash and speculative decoding for frontier-level inference performance, and Auto Endpoints to optimize model serving without manual tuning. For agentic workloads, they provide networked sandboxes with private IPv6, RDMA, and sidecars. Their capacity pool spans 17 cloud providers, giving them the elasticity to handle the 100,000-sandbox problem. Beyond inference, they also support serverless multi-node training for post-training and research, and even auto-research agents that launch GPU experiments.

For builders, the key insight is that AI has made infrastructure exciting again because it demands a fundamentally different stack. Operators should not force AI workloads into Kubernetes-shaped holes; instead, they should evaluate specialized primitives like Modal’s sandboxes for agent loops. Bubna also emphasizes that as agents write more code, observability becomes more important than reading code—a shift that will reshape how we debug and audit production AI systems. The move from developer experience to agent experience requires hard guardrails and specialized environments, not just faster GPUs.