All news

SARIA keeps agent KV caches across turns with coordinated DRAM and SSD reload to cut prefill latency.

vLLM fits long agentic contexts for GLM-5.3 on 16 H200s by splitting prefill and decode nodes and using prefill and decode context parallelism plus hybrid sparse attention.

vLLM serves DeepSeek-V4.1-Flash on one GB300 NVL72 rack at 954K tokens per second per GPU using prefill and decode disaggregation and wide expert parallelism, 7.9 times its single-node result.

Joint caching of large models with parameter sharing and inference offloading in mobile edge computing.

Shows routing to the most predictable model is suboptimal for LLM routing.

FairCache shows prefix and context caching makes multi-tenant LLM serving unfair and proposes cache-aware scheduling to fix it.

vLLM TorchTPU is a new backend that runs existing vLLM model code on Google Cloud TPUs without changes.

Proposes serving RAG workloads through decentralized collaboration of edge devices instead of the cloud.

vLLM supports MiniMax-M3.1-Flash on day 0, including its NVFP4 KV cache, NVFP4 MoE weights and a draft model for speculative decoding.

vLLM runs Recursive Flint, a 40B MoE with 2.5B active parameters, on one GPU at about 500 tokens per second with 100K context.