GLM-5.3 Optimizations, Part 2: PCP, DCP, and Hybrid HiSparse on 16 H200s
vLLM fits long agentic contexts for GLM-5.3 on 16 H200s by splitting prefill and decode nodes and using prefill and decode context parallelism plus hybrid sparse attention.
Extract
TL;DR: Long agentic contexts push GLM-5.3 against two limits on H200s: KV cache memory and prefill speed. We address both on 16 H200s, split into one prefill node and one decode node: Prefill Context Parallelism (PCP) splits each prompt across GPUs by token instead of by head, which suits GLM-5.3's sparse attention far better than tensor parallelism. Decode Context Parallelism (DCP) shards the prefill engine's KV cache by token, so a 1M context fits on the prefill node. Hybrid HiSparse lets decode keep KV in host memory, so a 1M-token request fits. On AgentX, PCP8 × DCP8 prefill with Hybrid HiSparse decode gives 1.7–3.2× the per-user speed of a data-parallel deployment at the same throughput, and it is the only configuration we tested that serves 1M-token contexts. Serving long contexts in memory-constrained deployments Agentic workloads have long contexts that grow with every turn, which puts pressure on both GPU memory and prefill. On H200s, GLM-5.3's weights take up most of each GPU's memory, leaving little room for the KV cache of those long contexts. And because every turn adds more tool output and file contents to prefill, under load the prefill engines can become the bottleneck for the whole deployment. Part 1 showed how Hybrid HiSparse lets long contexts fit in an aggregated deployment on a single 8× H200 node. Prefill/decode disaggregation takes more than that. On the prefill node, we use PCP to speed up long prompts by parallelizing across the prompt's tokens. With MLA, however, this leaves every PCP rank holding a full copy of the KV cache, so we combine it with DCP to remove that replication. On the decode node, Hybrid HiSparse allows higher concurrency at long contexts. vLLM's new modular core (Model Runner V2) makes composing all of this much easier. Why not TP or DP? Two standard ways to run GLM-5.3 prefill on an 8× H200 node are DP8 and TP8: DP8 assigns each prompt to a single GPU, so its attention cannot use the other seven. This limits time to first token (TTFT) for long prompts, and the prompt's KV cache must fit in that GPU's memory. TP8 partitions the attention heads across GPUs. Because MLA's latent KV is shared by all 64 heads, every GPU stores the full cache and reads it to serve only 8 heads. The indexer, which scores every earlier token to select the 2,048 that sparse MLA attends to, is not partitioned at all, so its quadratic cost is repeated on every GPU. Neither scheme spreads one long prompt's attention work across the eight …
Explained
What happened
The vLLM team (Red Hat, Mistral) serves GLM-5.3 with up to 1M-token contexts on 16 H200s, split into one prefill node and one decode node (P/D disaggregation). Three techniques:
- Prefill Context Parallelism (PCP8): split each prompt across 8 GPUs by token, not by attention head. The prompt is cut into 16 chunks and each GPU takes one from the front and one from the back, so causal-attention work is balanced.
- Decode Context Parallelism (DCP8): shard the KV cache round-robin by token across the 8 GPUs. A 1M-token request needs 53.4 GiB of FP8 KV but each GPU has only ~15 GiB free. With DCP it is 6.7 GiB per GPU.
- Hybrid HiSparse (decode): keep KV in host memory (48 GiB pool + 16 GiB offload per rank), so decode concurrency is bounded by CPU RAM, not HBM.
Results on SemiAnalysis AgentX (replayed Claude Code traces): 1.7 to 3.2x per-user speed vs data-parallel prefill at equal throughput, median TTFT down from 1.5-5.2 s to 0.8-0.9 s, and at concurrency 64 about 2x throughput (8,340 tok/s/GPU at 29.6 tok/s/user) while other setups fall to 1-2 tok/s/user. It is the only tested configuration that serves 1M contexts at all.
Why it works
GLM-5.3 uses MLA (multi-head latent attention): one compressed KV shared by all 64 heads. That breaks the usual scaling tricks:
- Tensor parallelism splits heads, but every GPU still stores the full shared KV and reads it to serve only 8 heads. That is about 26 FLOPs per byte, deeply memory-bound. The indexer (which picks the 2,048 tokens sparse attention looks at) is not split at all and is quadratic.
- PCP makes each KV row serve all 64 heads, raising intensity to about 191 FLOPs/byte, near the H200's compute-bound point (206). Each GPU indexes only its own tokens, so the indexer's quadratic cost stops being repeated 8 times.
- PCP alone still all-gathers full KV to every GPU, so memory becomes the limit beyond ~250K tokens. DCP fixes storage, at a cost of 6% to 15% extra per step for the per-layer all-gather.
The bigger pattern
Long-context serving is a memory-placement problem, not just a FLOPs problem. Pick the parallelism axis that matches the model's attention structure (tokens for MLA, heads for classic MHA) and use arithmetic intensity (FLOPs per byte moved) to judge it. Tiered KV (HBM, then host RAM) is now standard for agentic workloads whose context grows every turn.
What it means in practice
- Agentic AI workloads are prefill-heavy. If you design AI-serving systems, TTFT and KV capacity, not tokens/s, are usually the first bottleneck.
- The router matters: a prefix-aware router (
llm-d) plus CPU KV offload lets repeat turns skip prefill. It is the same idea as cache affinity in a sharded Redis tier. - Key flags to know in vLLM v0.32+:
--prefill-context-parallel-size,--decode-context-parallel-size,--enable-expert-parallel. - Learn to estimate bytes per token of KV for a model. It decides how many users fit per GPU, which is the core cost number for any AI product.