paperarXiv (systems, data, ML)·22 Oct 20265.4
SARIA keeps agent KV caches across turns with coordinated DRAM and SSD reload to cut prefill latency.
blogvLLM Blog·23 Oct 20265.2
vLLM fits long agentic contexts for GLM-5.3 on 16 H200s by splitting prefill and decode nodes and using prefill and decode context parallelism plus hybrid sparse attention.
paperarXiv (systems, data, ML)·22 Oct 20264.2
Re-selects KV cache visual tokens during long multimodal generation based on drift, under a budget.
paperarXiv (systems, data, ML)·23 Oct 20264.1
BeNice shares KV cache across loop iterations in looped transformers to close the inference memory gap.
paperIEEE Xplore·1 Nov 20264.0
FairCache shows prefix and context caching makes multi-tenant LLM serving unfair and proposes cache-aware scheduling to fix it.
paperarXiv (systems, data, ML)·22 Oct 20264.0
RIFT compresses draft-model KV cache for speculative decoding with complementary training.