Habitat is the online storage platform OpenAI built so products like ChatGPT can access data quickly and reliably. It now handles more than 70 million requests per second, stores over 500 petabytes, and supports products used by more than 1 billion people each week across nearly 40 geographic regions. Two years ago, Habitat was a simple Python client-side library connected to a single Azure Cosmos DB database; today it is a distributed service that also uses caching, Rockset, and Envoy.

The shift from library to service was driven by operational pain. As OpenAI‘s service count grew, backward-compatible protocol changes to the client library became infeasible and required coordinating deployments across dozens of services. The article describes a migration to regionally distributed Cosmos DB accounts that needed client-side routing logic behind a feature flag, shadowing to verify correctness, and a rollback of an unrelated service that reintroduced a buggy client and caused the very outage the team was trying to avoid. In response, OpenAI pulled Habitat into its own standalone service, giving the platform a single point of control for deployments, observability, and security enforcement, including access control, audit logging, and limiting direct access to underlying storage.

Running that service in Python was a deliberate, short-term tradeoff. Python added network latency, CPU overhead, and memory costs compared to library execution, and the team knew a rewrite would eventually be necessary. But the immediate priority was unblocking product developers and achieving platform stability. The main technical challenge became tail latency: because an average user request can result in hundreds of database calls, the slowest call is the one the user feels. The team found that asyncio scheduling delay can dominate p99 latency, since concurrency is not CPU parallelism and CPU-heavy work like routing, compression, encryption, checksumming, and hedging can leave ready responses waiting to be processed. They began measuring event loop scheduling delay in real time and responded by keeping each Python process serving only a small number of concurrent requests and scaling out far more worker processes.

Profiling uncovered one acute cause: Statsig was polling for refreshed feature flag configurations every minute with no jitter, and the configuration included every production rule across every service. With up to 8 Python processes per pod, all workers would periodically stall parsing this giant file. The fix was to deploy a smaller targeted config, lengthen the refresh interval, and add jitter to background tasks. Connection pooling created another failure mode. Client-side pooling, combined with aiohttp’s default LIFO connection reuse, led to a metastable failure: during bursts, slower overloaded servers returned connections to the pool later and therefore received even more requests, creating runaway degradation. Switching to FIFO reuse broke that feedback loop and reduced steady-state request variance. Today, OpenAI mostly relies on Istio and Envoy for connection pooling and load-aware balancing, and uses Envoy to multiplex HTTP/1 into HTTP/2 and centralize rate limits and circuit breakers.

Another reason Habitat could scale despite Python was its deliberately constrained API. Instead of allowing arbitrary SQL queries that could cause expensive table scans or joins, Habitat exposes a NoSQL API modeled on client-defined object and edge types, inspired by TAO. Clients can query direct edges of an object, but Habitat does not support general graph traversal. This keeps requests simple, predictable, and constant-work, making isolation and load balancing easier. For complex query needs, Habitat streams changes via change data capture to isolated Rockset instances, and each client team is responsible for scaling its own Rockset instance. The article frames this extra friction as a deliberate tradeoff that protects online storage from read-heavy analytical workloads.

Eventually the Python service reached its limit: at its peak it served more than 20 million requests per second, and it became the second largest service by core count at OpenAI. In Q2 2026, with just 2 engineers, Codex, and GPT‑5.5, the team rewrote the entire service in Rust. The Rust service now handles 95% of production requests and is 6x more CPU efficient and 15x more memory efficient than the Python version, with significantly lower average and tail latencies. Python will be deprecated in the coming weeks. The article is the first in a two-part series; the second plans to cover multi-tenancy reliability, read performance optimization, and how Habitat scaled with Azure Cosmos DB to handle the demand.