Serving DeepSeek-V4.1-Flash at Scale: vLLM Matches DeepInfer and Scales to GB300 NVL72
vLLM serves DeepSeek-V4.1-Flash on one GB300 NVL72 rack at 954K tokens per second per GPU using prefill and decode disaggregation and wide expert parallelism, 7.9 times its single-node result.
Extract
TL;DR: On one NVIDIA GB300 NVL72 rack, vLLM with P/D disaggregation and Wide-EP serves DeepSeek-V4.1-Flash on SemiAnalysis AgentX at 954K tokens/s per GPU at a P90 of 196 tokens/s per user, 7.9× our single-node result from the first post and $38.96 per GPU-hour at DeepSeek's off-peak prices. A specialized decode path reaches 826 tokens/s for a single user. Against DeepInfer, DeepSeek's own engine, vLLM's decode throughput is within 6% across batch sizes 8–256, and its prefill tracks DeepInfer's published curve. The efficiency comes from deliberate engineering on components that vLLM already has, built on the kernels DeepSeek open-sourced. This post walks through our performance study, breaks down each optimization, and shares the lessons we learned: A general-purpose engine can match a model-specific one Understand the model before optimizing Small DEP instances work best for prefill1 Wide-EP decode scales well with EPLB CPU overhead becomes critical at NVL72 scale A specialized decode path unlocks low latency Performance overview Figure 1: SemiAnalysis AgentX on NVIDIA GB300 NVL72: vLLM with P/D disaggregation and Wide-EP (gold) achieves a further 7.9× throughput increase over our aggregated single-node result from the first post. Throughput counts cached input, uncached input and output tokens (source). Compared to DeepInfer DeepInfer is DeepSeek's own inference engine for DeepSeek-V4.1-Flash, released this week as an offline engine with published prefill and decode curves. It is the most relevant reference for this model, so before going into our optimizations we show where vLLM stands against it. Decode. Under DeepSeek's decode configuration (131K context, nextn=3), vLLM on GB200 achieves throughput within 6% of DeepInfer's published results across batch sizes 8 to 256 (Figure 2). Figure 2: Decode output throughput per GPU vs. batch size at 131K context, nextn=3. vLLM on GB200 vs. DeepInfer's published results, read from the charts in its README (source). Prefill. DeepInfer's published prefill curve uses EP4DP4 with 32 redundant experts and a 140-token bounded replay, on hardware the chart does not state, and we read its values off the chart. Our run uses DEP2 on GB300 with no redundant experts, a 128-token replay and our own prompts. Because the setups differ, we do not read the gap between the two curves as an engine-to-engine speedup. What Figure 3 shows is that vLLM's DEP2 prefill tracks DeepSeek's own engine across the range they …
Explained
What happened
vLLM serves DeepSeek-V4.1-Flash on one NVIDIA GB300 NVL72 rack with prefill/decode disaggregation and Wide-EP (expert parallelism across many GPUs):
- 954K tokens/s per GPU at a P90 of 196 tokens/s per user on SemiAnalysis AgentX, 7.9x their single-node result, about $38.96 per GPU-hour at DeepSeek's off-peak prices.
- Decode throughput within 6% of DeepInfer, DeepSeek's own model-specific engine, across batch sizes 8 to 256.
- A specialized small-batch decode path hits 826 tokens/s for a single user on four B300s, 44% above the standard path.
Why it works
They measured the model before optimizing:
- KV cache is tiny: 890 bytes per token (cross-layer sharing plus FP4). So KV capacity rarely limits batch size.
- Decode is MoE-bound, not attention-bound: MoE is 62% of a step; going from 128K to 855K context adds only +7% latency.
That shaped every choice:
- Small prefill instances (DEP2). DEP4 lost 13.6% to data-parallel imbalance vs 5.9% for DEP2. Three kernel changes (bounded replay on layer 20, 2 MB GPU pages for host tables,
DeepSelecttop-k) gave +35.6% to +43.9% prefill throughput. - Wide-EP for decode, because MoE weight reads dominate. Spreading experts over 32 GPUs cut MoE time from 13.4 to 6.0 ms.
- EPLB (expert load balancer) rearranges experts every 3,000 steps. Balance went from 0.35 to 0.94 for the target model, and rebalancing all 32 ranks takes under a second over NVLink without pausing decode.
- CPU overhead fixes. An unoptimized DEP32 was slower than DEP16 (17.6 vs 13.8 ms/step) even though GPU work was faster. Fixes: async KV push via NIXL (141K to 440K tokens/s/GPU), a recursive-doubling CPU all-reduce (DP sync 10.8 to 1.5 ms), and capping admissions to 4 new KV loads per step.
The bigger pattern
At large scale, tail effects dominate: every rank syncs every layer, so the slowest rank sets the pace. Load balancing and host-side overhead matter as much as GPU kernels. Also: a general-purpose engine can match a model-specific one when it adopts that model's open kernels.
What it means in practice
- The same lessons apply to any fan-out system (Kafka consumers, sharded workers): measure imbalance directly with "mirror vs lag" style experiments, and fix the slowest participant first.
- Admission control (cap new work per tick) improved both throughput and P90. Worth copying in job queues and stream consumers that suffer from burst admission.
- Mixture-of-Experts serving cost is driven by weight reads, so wide expert parallelism and balancing are the main levers, not bigger GPUs.