Achieving near-zero-downtime scaling in Elastic Expert Parallelism

Traditionally, changing the number of GPUs in a vLLM service required adding or removing entire model replicas or restarting the engine with a different GPU count. As models grow larger, each replica needs more GPUs, so scaling by replica adds or removes GPUs in increasingly large increments. Restarting vLLM to scale DeepSeek V4 Pro from 16 to 20 GB200 GPUs stopped serving for 197 seconds (see the performance section for details), leaving those expensive GPUs allocated without generating tokens. In our first Elastic EP post, we showed how vLLM could add or remove GPUs from a running DEP deployment (data-parallel attention with expert-parallel MoE) without restarting it. While we avoided a restart, serving still stopped throughout reconfiguration as we initialized new workers, prepared the expanded communication groups, loaded model weights, and repeated kernel warmup and CUDA graph capture. In our DeepSeek V4 Pro benchmark, this pause lasted 100 seconds when scaling from 16 to 20 GPUs and 56 seconds when scaling back to 16 GPUs (full details in the performance section). In this post, we describe how asynchronous preparation and NIXL EP enabled Elastic EP reconfiguration with near-zero downtime. In the same benchmark, we reduced downtime to 0.82 seconds when scaling from 16 to 20 GPUs and 1.96 seconds when scaling back to 16 GPUs, allowing us to scale up and down with almost no impact on serving performance. Overlapping topology preparation with serving We reorganized vLLM's reconfiguration flow into two phases: background preparation and a short commit. Existing workers use separate background threads to prepare and warm up target communication, transfer model weights, and synchronize execution-related information (such as autotuning results) with joining workers, while continuing to serve requests with their current communication groups and CUDA graphs. In parallel, joining workers initialize, receive model weights, and prepare their communication and GPU execution. During the commit phase, vLLM briefly pauses its schedulers, activates the prepared topology, and resumes serving. These changes were introduced in asynchronous preparation and prepare/commit optimizations. Preparation covers four main tasks: Initialize joining workers. Start their processes and allocate storage for the model and execution. Prepare communication. Set up the target communication groups and expand the EP communicator. Transfer model weights. Transfer the required weights from …