How NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes

Training large-scale AI models on thousands of GPUs is a high-stakes operation where monitoring and automation must be built-in to optimize cluster resiliency and uptime at scale. This concept isn’t new to data center operators who have measured mean-time-to-repair (MTTR) and mean-time-between-failure (MTBF) long before AI factories existed. Now the stakes are higher. Modern AI factories run distributed training jobs across hundreds of nodes for days or weeks, and at this scale the operating model changes: hardware anomalies, software errors, network timeouts, and resource contention must be managed proactively to keep workloads on track. Traditional approaches that rely on manual intervention, static checkpointing, and reactive monitoring simply don’t scale when organizations are running dozens of these concurrent multi-node workloads in shared infrastructure. This is why software services, like the autonomous recovery engine offered in NVIDIA Mission Control, are critical to scaling production AI. Previously, the autonomous recovery engine, which can detect, diagnose, and recover from anomalies across the stack, was limited to high-performance computing (HPC) workloads leveraging Slurm. As a part of Mission Control 2.5, this service is now extended to Kubernetes-based workloads managed using NVIDIA Run:ai. This post demonstrates how this latest version of Mission Control brings its integrated resiliency framework to Kubernetes workloads, helping AI factory operations move from reactive operational maintenance to proactive, self-healing infrastructure. Built-in resiliency services The vision behind Mission Control’s autonomous recovery engine is simple: AI factories should be self-healing systems where failures are detected instantly, recovery decisions are made intelligently based on workload context and failure type, and jobs resume automatically with minimal disruption or wasted compute. The goal is to understand what failed, isolate anomalies and tag them for maintenance, and return workloads to productive compute with as little wasted time as possible. Figure 1. In an AI factory, recovery automation should move nodes out of risky or stranded states and back into productive, schedulable compute as quickly as possible to prevent lost work and missed opportunity costs There are two services that make up the autonomous recovery engine: autonomous job recovery autonomous hardware recovery Because true self-governing resiliency is best understood as a …