claude-news

Week of Monday, 19 October 2026

Serving huge MoE models gets cheaper, and old database structures get faster without being replaced

llm-serving

Serving giant MoE models: split by token, disaggregate, scale without restarts

What changed

The vLLM team published three deep posts in one week. On one GB300 NVL72 rack, vLLM serves DeepSeek-V4.1-Flash at 954K tokens/s per GPU, 7.9x its single-node result, and its decode is within 6% of DeepSeek's own engine. GLM-5.3 now serves 1M-token contexts on 16 H200s at 1.7 to 3.2x the per-user speed of a plain data-parallel setup. Elastic expert parallelism cut the pause when adding GPUs from 197 seconds (restart) to 0.82 seconds.

Key techniques

  • Prefill/decode disaggregation: separate pools for prompt processing and token generation, sized differently. Prefill runs on small groups (2 GPUs) to keep load balancing simple, while decode spreads experts wide across the rack because MoE decode is bound by weight reads.
  • Measure the model first: DeepSeek-V4.1-Flash needs only 890 bytes of KV per token, and 6.7x more context adds just 7% decode latency. So KV memory rarely limits batch size, and balancing can be by request count alone.
  • Prefill context parallelism: split a long prompt across GPUs by token instead of by attention head. Chunks are paired front and back so every GPU gets equal causal-attention work, which raises sparse attention from about 26 to 191 FLOPs per byte.
  • Decode context parallelism: shard the KV cache round-robin by token, so a 1M-token request needs 6.7 GiB per GPU instead of 53.4 GiB. The cost is a per-layer all-gather that adds 6% to 15% per step.
  • Prepare then commit: new GPUs load weights, warm up and connect in the background while old ones keep serving. A communication library that adds ranks in place keeps captured CUDA graphs valid, so the switch takes under 2 seconds.

Open questions

  • How much of this carries over to models without DeepSeek's tiny KV footprint, where KV memory does limit batch size?
  • Tiered KV caches on SSD (as in the SARIA paper, which cut SSD writes by 57% and TTFT by 42% to 80%) look like the next step for long agent sessions. Watch whether vLLM adopts selective persistence.
  • CPU overhead becomes a bottleneck at rack scale. Expect more work on moving scheduling off the critical path.
database-internals

Database internals: augment the B-tree, fix numeric, share tenancy at 20M QPS

What changed

This week's deeper database work shared one idea: make existing structures faster instead of replacing them. A PVLDB paper speeds up InnoDB B-tree lookups with a learned index on the side, without changing the tree. A Postgres deep dive explains why numeric aggregation is about 3x slower and uses 9x more memory than double precision. OpenAI described how its storage platform, now 500 PB and over 20M QPS on its largest Cosmos DB account, went from shared to dedicated containers and back to shared.

Key techniques

  • Augment, do not replace: MirrorBtr builds a learned model that predicts where a key sits in the B-tree, so lookups skip most of the traversal. The real tree stays the source of truth, which keeps transactions and recovery untouched.
  • Prefix pruning for set intersection: PRISM stores 64-bit integers in a trie of fixed-width prefixes, each node with a bitmap of occupied ranges. Whole ranges missing from one set are skipped without touching elements, and leaf segments map onto SIMD registers.
  • Fixed-width decimals: fast engines (DuckDB, ClickHouse, SQL Server) store decimals as plain integers with the scale in the column type. Postgres keeps scale in each value and computes result scale from the data, so values are variable-length and passed by reference.
  • Multi-tenancy with guardrails: throttle the hottest keys first, give bulk jobs a lower priority so interactive traffic wins, and split partitions on request load, not only storage. This removed the need for every team to plan capacity.

Open questions

  • Can learned side-indexes stay correct and cheap under heavy write workloads with frequent page splits?
  • Will Postgres ever get a fixed-width decimal in core? Three extension attempts stalled.
  • Failover at extreme scale took hours and the failback caused a connection stampede. How should clients behave so a region switch does not overload the gateway?