OpenAI‘s storage platform Habitat sits behind every OpenAI product, responding in milliseconds to more than 1 billion users across 40 regions and serving over 500 petabytes of data. Part I covered Habitat‘s evolution from a Python client to a service; Part II walks through scaling the underlying Azure Cosmos DB storage, including the multi-tenancy journey, reliability co-management with Azure, read-latency work, and change data capture (CDC) scaling.

Habitat initially relied on Azure Cosmos DB primitives with a single default container. Logical partition keys were exposed directly to clients, leading to collisions on commonly used keys like user_id or workspace_id. Because limits are enforced at the physical partition level, logical partitions could not actually reach the documented 10K RU/s, and noisy neighbors could disrupt unrelated workloads such as login and signup flows. Weak partitions made this worse: as data grew and physical partitions split, total provisioned throughput stayed flat while per-partition throughput shrank, reducing capacity for storage-heavy use cases.

To escape these problems, OpenAI moved critical use cases into separate containers. This reduced blast radius and avoided many site-wide outages, but it pushed capacity planning onto product teams. A use case going from 0 to 1 million QPS could represent 1,000x more load than its internal dogfooding baseline, and teams often forgot to pre-scale before launch. With help from the Azure team, Habitat shifted back to one shared container by default, using hot key throttling, request tiering QoS for offline workloads, and a new serverless mode that allows 10K RU/s per physical partition and RU-usage-based splits. Since then, OpenAI reports no provisioning or noisy neighbor incidents.

At OpenAI‘s scale, managed services become less managed. OpenAI and Azure now jointly monitor Habitat from service layers down to Azure Cosmos DB physical storage, test against a production-sized non-production account, and have direct paging channels. In late 2024, a regional outage showed that large accounts would take hours to fail over; a subsequent failback triggered a connection stampede caused by gateway-mode client failover behavior. Joint work on control-plane and gateway scalability, connection pooling, and isolating critical workloads from offline workloads in separate VPCs has been central to stability. The largest Azure Cosmos DB account stores 10s of petabytes and serves more than 20M QPS.

Reads represent over 90% of Habitat requests, and compute and storage capacity were often in different regions. A layered strategy improved global read p50 from roughly 120 ms to roughly 10 ms, a 12x improvement: colocating data near users where physical capacity allows, broadly replicating only the smallest and most latency-sensitive datasets such as authentication metadata, and integrating a distributed lookaside cache using Valkey with invalidation driven by Azure Cosmos DB‘s change feed, following approaches like Meta’s TAO. The Habitat service itself was rewritten in Rust to reduce proxy overhead.

The same CDC infrastructure captures millions of changes per second across thousands of containers and millions of partitions, feeding more than a hundred Kafka topics for secondary indexes, data movement, offline analytics, and cache invalidation. An internal control plane distributes work among thousands of workers, uses replica regions and lower-priority feed reads to protect interactive traffic, and switches readers during regional outages. Dead letter queues prevent head-of-line blocking, and the team modified Azure’s CDC SDK to batch checkpoints in memory.

Looking ahead, OpenAI is building a custom in-house exabyte-scale online storage system for bulky Habitat data types. It combines the RocksDB Cloud core from Rockset with Raft-based replication, implements the same document model as Azure Cosmos DB, and uses disaggregated storage and physical replication to reduce flash capacity at the cost of lower peak QPS.