SARIA: Persistence-Aware KV Caching with Coordinated DRAM--SSD Reload for LLM Agents
SARIA keeps agent KV caches across turns with coordinated DRAM and SSD reload to cut prefill latency.
Paper (from PDF)
Read full textSARIA: Persistence-Aware KV Caching with Coordinated DRAM–SSD Reload for
LLM Agents HuiLi1,QihuiZhou2,XiaoYan3,ShumengZhang4,ZijieDai1 ZequnGong2,MingcongSong2,GuopingLong2,HongjieSi2,JamesCheng1 1TheChineseUniversityofHongKong 2Huawei 3WuhanUniversity 4XiamenUniversity 6202 tcO 22 ]SO.sc[ 1v96113.0162:viXra Abstract SARIA --- --- -------- --- --- --- --- --- --- ----- --- --- --- Multi-turnLLMagentsusuallyreuselongcontextprefixes, 3M 8M 16M 32M 64M 3M 8M 16M 32M 64M --- --- --- --- --- --- --- --- ------ ------- --- ------ ------- makingkeyvalue(KV)cachingcrucialforreducingprefill tegraT ecnarudnE latency.SSDsprovidealow-costextensiontolimitedDRAM 98% of max HR 4.5-monlifetime capacitybutintroducetwokeychallenges. Eagerlypersist- ------------------------------------- --- --- --- --------------- --- --- ---- ---------------- --- ---- --- --- 1.5M 11×for endurance 1.5M ingallKVcacheblocksincurssubstantialwritetrafficand SARIA enables rapidlyconsumesfiniteSSDendurance.Meanwhile,thesmall Capacity:3M➔32M HR:82%➔84% lifetime-HRtrade-offs --- --- --- --- --- --- --- --- ---------- --- --------------------- --- --- numberofSSDsprovisionedintypicalGPUserverslimits aggregatereloadbandwidth,makingSSD-residentKVcache much slower to retrieve than that in DRAM and inflating ----------- ----------- --------- ------- --- --------- --- --- --- --- --- --- --- prefilllatency. Figure1: P90cachehitrateandestimatedSSDlifetimefor --------- ------------------------------------- --- --- --- --- -------- ----------------------------------------- --- --- --- --- --- Wepresent SARIA …
Explained
What happened
SARIA (CUHK, Huawei and others) is a KV-cache system for multi-turn LLM agents that uses SSDs as a cheap tier below DRAM, without burning out the SSDs or stalling on them. Built on LMCache, tested on TraceLab and WEKA agent traces with full-attention and hybrid-attention models:
- 57.1% to 58.9% fewer SSD writes than the common "copy everything to SSD" approach.
- 42.3% to 79.7% lower mean time to first token (TTFT) than DRAM-only caching.
- 26.7% to 31.9% faster reloads for the slowest quarter of requests.
Why it works
Agents resend long, stable prefixes every turn (system prompt, tool schemas, repo instructions, history). Caching their KV (the attention state computed during prefill) lets the next turn skip prefill. HBM and DRAM are too small, so systems like Mooncake and LMCache spill to SSD. That causes two problems:
- SSD endurance. Writing every new KV block to SSD ("full duplicate caching") is brutal. At vLLM's 12,963 prefill tokens/s per GPU for GLM-5.2, one GPU writes about 1.1 GiB/s, or 103.7 TB/day. A 4 TB drive rated at 1 drive-write-per-day uses its 5-year budget in ~70 days. Buying bigger SSDs for endurance barely helps hit rate (82.0% to 84.8%).
- SSD bandwidth. A GPU server has few NVMe drives (8 for 8 H100s on AWS p5), while matching one x16 GPU link needs about 4 SSDs. Reloads from SSD are slow.
SARIA's fixes:
- Selective admission (write path): persist a block to SSD only if history says it will be reused after it falls out of DRAM (it was evicted before and then needed again), or if it is very popular. Blocks that live and die in DRAM never touch the SSD.
- Lazy partial promotion (read path): after an SSD hit is loaded into the GPU via GPUDirect Storage, copy only some of those blocks into DRAM. Next time, the reload is split across DRAM and SSD in parallel without crowding DRAM.
- Transfer coordinator: slow DRAM reads down while SSD reads run, so both finish together instead of DRAM hogging GPU ingress and leaving a long SSD tail.
The bigger pattern
This is classic tiered caching with admission control, the same idea as TinyLFU or "don't cache one-hit wonders" in CDNs: the cheapest write is the one you skip. Flash wear makes write amplification a cost metric, not just a speed metric.
What it means in practice
- For AI serving at scale, KV-cache tiering (HBM, DRAM, SSD) is becoming core infrastructure, much like Redis plus Postgres in web stacks.
- Track endurance (DWPD) as a capacity-planning metric for any write-heavy SSD cache, including Kafka and log storage.
- Admission policies based on reuse history (evicted-then-requested) are a cheap, general trick for any multi-tier cache you build.