Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)
Part 2 of how OpenAI scaled its storage platform Habitat to serve over 1 billion ChatGPT users across 40 regions with millisecond latency.
Extract
All OpenAI products require fast, reliable access to data, provided by our storage platform, Habitat. It has to respond in milliseconds to over 1 billion users across 40 geographic regions. Our infrastructure faced an extraordinary challenge: supporting explosive growth and new use cases while continuing to improve reliability, performance, and product developer velocity. Solving that challenge required rethinking how our online storage platform was designed and operated. We constantly evaluated tradeoffs, following the pattern of: first become reliable enough for mission-critical product traffic, then fast enough for global users, and, finally, to deftly operate at massive scale.This two-part series explains how we developed Habitat from a simple client library and database into a globally distributed storage platform serving over 500 petabytes of data. Part I focused on Habitat’s evolution from a Python client to a service. It also covered the techniques that allowed us to push a Python-based service far beyond what many would consider practical.In this post, Part II, we walk through how we scaled the underlying storage component: Azure Cosmos DB. Through a close working partnership with the Azure team, we arrived at a viable multi-tenancy approach of co-managed containers, with excellent read performance at massive, global scale.Figure 01 · What is Habitat?Online storage platformHabitat is the online storage platform we built so OpenAI products can quickly and reliably access needed information.RequestResponseChanges (CDC)Making multi-tenancy reliable at scaleAs companies scale, infra teams often evolve from running single-tenant systems to multi-tenant ones. Single-tenancy, where every product surface team maintains their own system, reduces blast radius and provides easy ownership boundaries and workload isolation. As teams and product surfaces grow, this becomes expensive to run. Single-tenancy makes every product team responsible for capacity planning and often results in significant over-provisioning.Conversely, multi-tenancy assigns control to a centralized database team, with better knowledge and experience for provisioning resources. But multi-tenancy creates its own problems, such as noisy-neighbor issues, where one tenant consumes an unusually large share of the resources to the detriment of the other tenants.In Habitat, we had to weigh all these tradeoffs to design a reliable, efficient, storage system at scale. Today, Habitat is a …
Explained
What happened
OpenAI's Part II on Habitat, its online storage platform: 500+ PB, 1B+ users, 40 regions, built on Azure Cosmos DB. The largest account holds tens of petabytes and serves 20M+ QPS. Three stories:
- Multi-tenancy, then single-tenancy, then back.
- Phase I: one shared container. Problems: clients picked partition keys directly, so
user_idcollided across products. Noisy neighbors on one physical partition broke login and signup. Weak partitions: as data grows, partitions split but total throughput stays fixed, so each partition gets less. - Phase II: a container per service. Isolation worked, but every team now had to do capacity planning, and launches went from 1K RU/s in testing to 1M RU/s on day one, causing SEVs.
- Phase III: back to shared by default, with hot-key throttling, QoS tiers (offline jobs throttled first), and a new serverless mode with RU-based splits (up to 10K RU/s per partition). No noisy-neighbor incidents since.
- Phase I: one shared container. Problems: clients picked partition keys directly, so
- Co-managed service. A 2024 regional failover took hours, and the failback caused a connection stampede on the gateway. Now both teams share dashboards, test on a production-sized non-prod account, and can page each other directly.
- Global read latency: p50 from ~120 ms to ~10 ms (12x). Replicate only tiny hot datasets (auth metadata) to all 40 regions, use a Valkey lookaside cache near users invalidated by CDC from the change feed (TAO-style), and rewrite the proxy from Python to Rust.
Why it works
- Mixing workloads smooths aggregate growth. In a shared pool one launch is a rounding error; in a dedicated pool it is a 1000x spike. Isolation should come from throttling and priorities, not from separate hardware.
- CDC-driven invalidation beats TTLs: TTLs force a choice between staleness and hit rate, while invalidation from the change feed gives both within seconds.
- Reads are 90%+ of traffic with a small hot working set, which is exactly where caching hides cross-region latency.
The bigger pattern
This is the classic blast radius vs utilization trade-off of multi-tenant storage, plus the lesson that at extreme scale managed services stop being managed. CDC (millions of changes/s into 100+ Kafka topics) becomes a platform of its own, with dead letter queues to avoid head-of-line blocking and batched checkpoints.
What it means in practice
- Never let clients choose raw partition keys. Namespace keys per use case (
product:user_id) so hot keys don't collide. - For a shared Postgres/Redis serving many features: add per-tenant rate limits and priority lanes (interactive vs batch) before you split databases.
- The CDC-invalidated cache pattern works at any size: Postgres logical replication or Debezium, into Kafka, invalidating Redis keys. Use DLQs so one poison message doesn't block a partition.
- Test failback, not just failover. Stampedes happen on the way back.