newsInfoQ·27 Oct 2026·Khawaja Shams
Presentation: Serving LLMs at Scale: The Hidden KV Cache Advantage
A talk on scaling LLM serving from a distributed systems view: GPU utilization, KV cache memory cliffs, tiered cache storage and tail latency.
Extract
Khawaja Shams shares a distributed systems perspective on scaling LLM inference stacks efficiently. He discusses practical strategies for optimizing GPU utilization, managing KV cache memory cliffs, leveraging tiered storage, reducing tail latency, and implementing disaggregated prefill and decode architectures to improve overall fleet performance, cost economics, and availability.
By Khawaja Shams