Inside Prometheus: Meta’s Journey to a Gigawatt-Scale Supercluster
Meta describes the first phase of Prometheus, its gigawatt-scale training supercluster behind the Muse models: power, networking and how it scales across buildings.
Extract
With the recent release of the Muse family of foundational models, we want to take you inside the infrastructure that has made it all possible. The first phase of Prometheus, our gigawatt-scale supercluster, is already complete and we’re working toward scaling it to comprise ~1 million GPUs. Prometheus also features technologies like closed-loop liquid cooling to help it operate efficiently. Right now, in New Albany, Ohio, Meta is undertaking its most ambitious AI infrastructure project to date. Prometheus, our gigawatt-scale supercluster, is an advancement in our infrastructure that is training the next generation of foundational models that will bring new AI experiences to billions of people. Two years ago, New Albany was a conventional data center site running web traffic. With advancements in hardware, networking and building design, we reworked this site to be home to training programs for frontier models. We’re building the supercluster in phases. The first phase has already been completed and was used to train our Muse foundational models , including the Muse Wonder models, our new frontier models that deliver state-of-the-art performance on personal and professional agentic tasks. Large training jobs require everything to work optimally to sustain their performance; an issue on a single host can slow down or stop the entire job, lowering its ‘goodput’. By focusing on improving our systems’ stability and reliability we were able to achieve more than 85% goodput for pre-training runs and more than 90% goodput for reinforcement learning (RL) runs of Muse Wonder. During these training runs, we were able to achieve an overall wall-clock mean time between interruptions (MTBI) of ~47k GPU hours. Since then, we have landed further changes across our fleet including firmware fixes to increase MTBI to more than 54K GPU hours, and we are constantly improving it. Now, we’re continuing to scale Prometheus with our next frontier models from Meta Superintelligence Labs in mind. https://engineering.fb.com/wp-content/uploads/2026/10/AIM-1210-Superclusters_Video_v1.mp4 The Prometheus supercluster is housed in Meta’s New Albany, Ohio, data center. The Supercluster: Unlocking New AI Capabilities Today, Prometheus comprises an array of hundreds of thousands of GPUs across multiple interconnected buildings, including traditional and AI-optimized data center buildings and rapid deployment structures, all tightly interconnected by miles of fiber and functioning as a …
Explained
What happened
Meta published the design of Prometheus, the training supercluster in New Albany, Ohio, that trained its Muse models.
- The first phase holds hundreds of thousands of GPUs across several buildings, including tents and quick-build structures, linked by fiber so they act as one machine. The target is about 1 million GPUs.
- Two rack platforms are live. Catalina uses
GB200 NVL72in a two-rack variant with 800G of network per GPU, chosen early because the default 400G would have starved a much faster GPU. Clemente usesGB300 NVL72in a single rack with 288 GB of HBM per GPU. - A separate Ethernet back-end network carries GPU-to-GPU traffic. Each building fabric connects up to 20,736 GPUs. An aggregation layer links buildings 2 km to over 100 km apart.
- Training reached over 85% goodput in pre-training and over 90% in RL, with a mean time between interruptions of about 47k GPU hours, later raised past 54k with firmware fixes.
- Each 72-GPU rack runs a 64-GPU slice and keeps 8 GPUs as hot spares. A 32,768-GPU job therefore provisions 523 racks, about 13% more GPUs than it uses.
Why it works
Synchronous training is only as fast as its slowest part. One bad GPU or link stalls the whole job. Meta attacks this in three ways:
- Match the network to the parallelism. Tensor and pipeline traffic, which is latency sensitive, stays inside a rack or a building. Only data-parallel gradient traffic, which tolerates delay, crosses the long links between buildings.
- Spread big flows. Training sends a few huge, long flows that collide under normal hashing. The first fabric sprayed packets across paths and reordered them in the switches. Newer NICs reorder at the endpoint, so cheaper shallow-buffer switches with adaptive routing now do the job.
- Spares inside the failure domain. NVL72 makes the blast radius 9x bigger than the old 8-GPU nodes. Because parallel groups like powers of two, 8 of 72 GPUs are free anyway. Using them as spares lets a job restart from checkpoint at once instead of waiting for a repair.
The bigger pattern
This is goodput engineering: at scale, the fraction of time doing useful work matters more than peak FLOPS. The ideas are known from HPC (topology-aware placement, spare nodes, checkpoint restart). What is new is the size of the failure domain, the distances, and turning a geometric leftover (64 of 72) into the reliability budget for free.
What it means in practice
- Measure goodput, not utilization. Track interruptions per hour and time lost per restart, the same way you would track error budgets.
- Put chatty traffic close together and send only delay-tolerant traffic over slow links. The same rule applies to services: keep synchronous calls in one zone and push cross-region work through queues or replication.
- Plan spare capacity inside the unit you allocate, not only outside it. A Kafka or Redis cluster that can lose one node without a rebalance follows the same idea.
- Expect many large flows to collide on hashed paths. If replication or backfill traffic causes hot links, look at flow spreading before buying bandwidth.