How NVIDIA Minerva Turns Telemetry Chaos Into Peak GPU Yield

AI Factories are some of the most technologically complex operational environments ever created. Hundreds of thousands of GPUs, NVIDIA Spectrum-X Ethernet scale-out, scale-across and scale-in fabrics, NVIDIA NVLink scale-up fabrics, distributed schedulers, and multi-week training runs operate in continuous lockstep.These layers function as one production system even when different teams own their telemetry. In synchronous distributed training, a job advances at the pace of its slowest rank. One degraded GPU, interface, storage device, or switch path can waste useful time across many healthy resources. The failures that hurt yield are often subtle: A GPU runs a few degrees hotter than its peers. An NVMe device loses performance only under sustained load. A switch generates a pause-frame storm that looks like dozens of unrelated symptoms. None of these signals necessarily crosses a static threshold. Together, they create stragglers, extended step times, and delayed checkpoints. Diagnosis becomes harder because the evidence is split across teams. Network operators see interface counters and fabric events. Compute operators see node and GPU health. ML teams see changing step times, failed jobs, or unstable inference latency. Each view is accurate, but the relationships between them remain hidden. Triage becomes a chain of dashboard searches and cross-team handoffs, which can turn a contained fault into hours or days of lost production time before the affected node or fabric path is identified. This post covers how NVIDIA Minerva, an AI-native AIOps platform designed around AI factory topology and workloads, addresses these operational gaps. NVIDIA Minerva collects telemetry across the stack, maps infrastructure to active jobs in a dynamic knowledge graph, applies purpose-built machine learning, and correlates dependent symptoms into high-fidelity incidents. An AI agent-powered assistant and MCP server connected to external agentic tools give operators a conversational path through the same live evidence. The goal is practical: preserve effective GPU hours and keep model training moving. Why conventional monitoring breaks at AI factory scale Rule-based monitoring assumes that a known metric crossing a known boundary indicates a known condition. That model works for clear failures. It struggles with so-called “gray failures,” where a component stays available, and the relevant issue may appear only under load or in comparison with peers. Absolute thresholds can …