Meta has published a detailed look at Prometheus, its gigawatt-scale AI supercluster in New Albany, Ohio. Phase one is complete, has already trained the Muse foundation model family including Muse Wonder, and Meta plans to scale the site toward roughly one million GPUs. The same campus was a conventional web data center only two years ago.

Meta also disclosed strong reliability data. It reports more than 85% goodput on Muse Wonder pre-training and over 90% on RL runs, with a wall-clock mean time between interruptions near 47,000 GPU hours during those jobs; firmware fixes have since pushed MTBI past 54,000 GPU hours. Meta’s target is two interruptions per 8,000 GPUs per day, or about 100,000 GPU hours between outages.

Two NVL72 Generations in Production, Vera Rubin Next

Prometheus runs two NVIDIA GPU platforms today, both built on NVL72 where 72 GPUs share one NVLink domain; a third platform based on Vera Rubin NVL72 is coming. The first, Catalina, uses GB200 NVL72 in the Ariel variant: a two-rack design with 36 GPUs per rack on Meta’s Open Rack V3 high-power specification. Each Blackwell GPU pairs with a Grace CPU and two ConnectX-7 400G NICs, giving every GPU 800G of scale-out bandwidth.

Meta chose Ariel because GB200 NVL72 scaled FLOPS, HBM bandwidth, scale-up bandwidth, and domain size well beyond H100. A Catalina pair delivers 72 Blackwell GPUs, 13.4TB of HBM3e, 720 PFLOPS peak FP8 compute, 130TB/s of NVLink bandwidth, 576TB/s of aggregate memory bandwidth, and about 120kW across two racks.

The second platform, Clemente, uses GB300 NVL72. Per-GPU HBM rises 1.5x to 288GB, dense FP4 throughput increases similarly, and attention acceleration roughly doubles. ConnectX-8 SuperNICs deliver 800G per GPU natively, while a 2:1 GPU-to-CPU ratio fits the full 72-GPU domain in one rack. Clemente offers 20.7TB of HBM3e, over an exaFLOP of peak FP4 compute, 130TB/s of NVLink bandwidth, about 135kW per rack, and a 5–10% performance gain over GB200 NVL72.

Meta’s next platform, San Miguel, will bring Vera Rubin NVL72 into the same fabric, with 288GB HBM4 per GPU at 19.2TB/s, 2.52 exaFLOPS peak FP4 compute, 216TB/s NVLink bandwidth, about 1,500TB/s aggregate memory bandwidth, fully liquid-cooled trays above 170kW per rack, and 10–20% better performance per watt than GB300 NVL72.

Dedicated Ethernet Backend Across Buildings

Training traffic runs on a dedicated Ethernet backend separate from frontend service traffic. Latency-sensitive collectives stay inside building fabrics, while more tolerant inter-replica traffic crosses an aggregation layer linking buildings. Meta first used this hierarchy on smaller clusters before extending it to Prometheus.

Each building fabric supports up to 20,736 GPUs. Meta moved from a Disaggregated Scheduled Fabric to a Non-Scheduled Fabric using shallow-buffer 51.2T switches, Minipack3 and Minipack3N, and Meta’s FBOSS stack. Above the buildings is the Backend Aggregation network, a multi-layer hub-and-spoke design on Arista 7800-series chassis with Broadcom Jericho3 silicon, spanning 2km to over 100km and delivering multiple petabits per second between regions.

64 of 72 GPUs Do the Work

Moving from H100’s 8-GPU NVLink domains to NVL72 multiplied the failure blast radius by nine, so Meta plans capacity around the 72-GPU rack. Each NVL72 domain runs a 64-GPU active slice and reserves eight GPUs as in-rack spares. For whole-rack faults, Meta keeps about 2% extra racks in reserve.

Two recovery paths handle faults. Isolated replica failures use parallelism-aware fault tolerance, while larger faults roll the job back to the last checkpoint. Meta checkpoints frequently, saving model state asynchronously to multi-tier storage with delta checkpointing and in-memory staging.

99% Drop in Interruptions

Prometheus was not stable at first. In mid-2025, interruption rates peaked near 1,000 events per 8,000 GPUs per day; a year of work cut that by 99%. Meta credits co-engineering with NVIDIA, on-site triage, continuous telemetry, firmware automation, improved memory error handling, and millisecond-scale switch health monitoring.

Power Transients and Mixed Cooling

Synchronous training swings cluster load by 30–40%, and large job starts or stops can shift tens of megawatts in seconds. Meta’s Dynamo Dimmer gradually caps GPU power near electrical limits, while rack batteries, capacitor trays, facility storage, and utility coordination absorb transients.

Cooling varies across buildings due to rapid expansion. Older spaces use modular air-assisted liquid cooling; the AI-optimized building uses closed-loop facility liquid cooling, which Meta calls the most efficient approach for GPU servers.

Hyperion Next

Meta will expand Prometheus with GB300 NVL72 and Vera Rubin NVL72 racks, while lessons from New Albany inform Hyperion, a new supercluster in Richland Parish, Louisiana, with closed-loop liquid cooling, full Entergy grid connectivity, and next-generation networking. Meta says it pays full energy costs at both sites and matches usage with new clean renewable generation.

The key operational takeaway is that at this scale, hardware is only the starting point. The real work to make hundreds of thousands of GPUs usable lies in networking, fault tolerance, and telemetry.

Beijing Qianxing Jietong Technology Co., Ltd.
Sandy Yang/Global Strategy Director
WhatsApp / WeChat: +86 13426366826
Email: yangyd@qianxingdata.com
Website: www.qianxingdata.com/www.storagesserver.com
Business Focus:
ICT Product Distribution/System Integration & Services/Infrastructure Solutions
With 20+ years of IT distribution experience, we partner with leading global brands to deliver reliable products and professional services.
“Using Technology to Build an Intelligent World”Your Trusted ICT Product Service Provider!