← Back to summary
SARIA: Persistence-Aware KV Caching with Coordinated DRAM--SSD Reload for LLM Agents
Converted from PDF with markitdown. Layout may be rough.
SARIA: Persistence-Aware KV Caching with Coordinated DRAM–SSD Reload for
| | | | | | LLM | Agents | | | | | | |
| --- | --- | --- | --- | --- | --- | ------ | --- | --- | --- | --- | --- | --- |
HuiLi1,QihuiZhou2,XiaoYan3,ShumengZhang4,ZijieDai1
ZequnGong2,MingcongSong2,GuopingLong2,HongjieSi2,JamesCheng1
1TheChineseUniversityofHongKong 2Huawei 3WuhanUniversity 4XiamenUniversity
6202 tcO 22 ]SO.sc[ 1v96113.0162:viXra
| | | Abstract | | | | | | | SARIA | | | |
| --- | --- | -------- | --- | --- | --- | --- | --- | --- | ----- | --- | --- | --- |
Multi-turnLLMagentsusuallyreuselongcontextprefixes,
| | | | | | | | 3M | 8M 16M | 32M 64M | 3M | 8M 16M | 32M 64M |
| --- | --- | --- | --- | --- | --- | --- | --- | ------ | ------- | --- | ------ | ------- |
makingkeyvalue(KV)cachingcrucialforreducingprefill
tegraT ecnarudnE
latency.SSDsprovidealow-costextensiontolimitedDRAM 98% of max HR
4.5-monlifetime
| capacitybutintroducetwokeychallenges. | | | | Eagerlypersist- | | | | | | | | |
| ------------------------------------- | --- | --- | --- | --------------- | --- | --- | ---- | ---------------- | --- | ---- | --- | --- |
| | | | | | | | 1.5M | 11×for endurance | | 1.5M | | |
ingallKVcacheblocksincurssubstantialwritetrafficand
SARIA enables
rapidlyconsumesfiniteSSDendurance.Meanwhile,thesmall Capacity:3M➔32M
| | | | | | | | | HR:82%➔84% | | lifetime-HRtrade-offs | | |
| --- | --- | --- | --- | --- | --- | --- | --- | ---------- | --- | --------------------- | --- | --- |
numberofSSDsprovisionedintypicalGPUserverslimits
aggregatereloadbandwidth,makingSSD-residentKVcache
| much slower | to retrieve | than that | in DRAM | and | inflating | | | | | | | |
| ----------- | ----------- | --------- | ------- | --- | --------- | --- | --- | --- | --- | --- | --- | --- |
prefilllatency.
| | | | | | | Figure1: | P90cachehitrateandestimatedSSDlifetimefor | | | | | |
| --------- | ------------------------------------- | --- | --- | --- | --- | -------- | ----------------------------------------- | --- | --- | --- | --- | --- |
| Wepresent | SARIA (SelectiveAdmissionandReloadfor | | | | | | | | | | | |
GLM-5.2(1DWPD)asSSDcapacityvaries(labelsdenoteto-
| Inference | Acceleration),a | KV caching | system | | that jointly | | | | | | | |
| --------- | --------------- | ---------- | ------ | --- | ------------ | --- | --- | --- | --- | --- | --- | --- |
kenslots,e.g.,1.5M).SARIA(right)canextendSSDlifetime
| manages | DRAM and | SSD by exploiting | | the session-level | | | | | | | | |
| ------- | -------- | ----------------- | --- | ----------------- | --- | --- | --- | --- | --- | --- | --- | --- |
relativetofullduplicatecaching(left)atthesamecapacityby
temporallocalityofLLMagentstoreduceSSDwritesandac-
tradingcachehitrateforlongerlifetime.
celeratecachereload.Onthewritepath,SARIAuseseviction
andreusehistorytoselectivelypersistKVcacheblocksthat
exhibit reuse opportunities beyond theirDRAM residency, schemas, repository instructions, and conversation history
avoidingunnecessarywritesforKVcachealreadyhostedin
oftenconstitutelongprefixesthatrecurthroughoutanagent
DRAM.Onthereadpath,SARIApromotesselectedSSD-hit session. Retaining the KV cache of these prefixes allows
KVcacheblocksfromGPUmemoryintoDRAMafterprefill,
| | | | | | | subsequent | | requests to | avoid repeated | | prefill computation, | |
| --- | --- | --- | --- | --- | --- | ---------- | --- | ----------- | -------------- | --- | -------------------- | --- |
allowingsubsequentreusewithinthesamesessiontoreduce
reducingtimetofirsttoken(TTFT)[9,20,32,39,43].However,
SSDreadtrafficwhilelimitingDRAMcontention.Atrans- GPUHBMandhostDRAMprovidelimitedcapacityathigh
fercoordinatorpacesDRAMreadswhileSSDtransfersare
cost,makingitdifficulttoretainallreusableKVcacheblocks.
active,allowingbothpathstomakeprogressandefficiently SSDs offera storage tierwith largercapacity and lower
shareGPUingressbandwidth.
| | | | | | | cost. | Existing | systems | such as | Mooncake | [22] | and LM- |
| ------------ | ----- | ---------- | --- | --- | ----------- | ----- | -------- | ------- | ------- | -------- | ---- | ------- |
| We implement | SARIA | on LMCache | MP | and | evaluate it | | | | | | | |
Cache[3]havealreadyextendedKVcachehierarchiesfrom
usingworkloadsfromTraceLabandWEKA,coveringboth
| | | | | | | host DRAM | | to SSDs. | Meanwhile, | I/O | technologies | such |
| --- | --- | --- | --- | --- | --- | --------- | --- | -------- | ---------- | --- | ------------ | ---- |
full-attentionandhybrid-attentionmodels.Theresultsshow
| | | | | | | as GPUDirect | | Storage | (GDS) | [18] enable | direct | SSD-to- |
| --- | --- | --- | --- | --- | --- | ------------ | --- | ------- | ----- | ----------- | ------ | ------- |
that SARIA reduces SSD writes by 57.1–58.9% compared GPU data transfer at high efficiency. With sufficient SSD
withpopularfullduplicatecachingandthemeantimetofirst
parallelismandanoptimizedI/Ostack,recentworkshows
token(TTFT)by42.3–79.7%overDRAM-onlycaching.
| | | | | | | that SSD-based | | KV caching | can | approach | DRAM | perfor- |
| --- | --- | --- | --- | --- | --- | -------------- | --- | ---------- | --- | -------- | ---- | ------- |
mance[23].TheseadvanceshighlightthepotentialofSSD-
basedKVcachingbuttwochallengesneedtobetackledwhen
1 Introduction
deployingSSDcachingontypicalGPUservers.
AgenticLLMservicesareturningthekeyvalue(KV)cache Challenge 1: Finite SSD endurance. SSDs can only sus-
fromephemeral,per-requeststateintoreusableservingstate tainafinitecumulativewritevolumebeforeexhaustingtheir
thatspansmultipleinteractionturns. Systemprompts,tool ratedendurancebudget[16].Existingsystemsduplicateall
1
newlygeneratedKVblocksfromDRAMtoSSD,whichwe butcontentionfromotheractivesessionsorlonginter-request
callfullduplicatecaching.Assuch,writetrafficscaleswith intervalscanevicttheseblocksbeforetheyarereused.There-
themodel’sKVfootprintandprefillthroughput.Forexam- fore,repeatedlyevictedKVcacheblockshavehighpotential
ple,vLLMreports12,963prefilltokens/sperGPUforGLM- forreuseafterDRAMeviction.Wealsopersistexceptionally
5.2 with 8K-token inputs on B300 GPUs [5]. At this rate, popularKV cache blocks (basedon theirreuse counts) be-
eachGPUwouldgenerateapproximately1,145MiB/sofKV foretheyareevicted. SARIA allowsuserstotradeoffSSD
writes,or103.7TB/day.A4-TBSSDratedatonedrivewrite writevolumeagainstcachecoveragebyconfiguringeviction
perday(DWPD)wouldconsumeitsfive-yearwritebudgetin andreuse-countthresholds(§4.2).SARIAfurtherdesignsuni-
about70daysofcontinuousoperation. fiedcontextmetadatatosupportbothclassicalfull-attention
Increasing SSD capacity provides a larger write budget, modelsandemerginghybrid-attentionmodelsandplansvalid
but the capacity needed for endurance can far exceed that prefixreloadsfrombothDRAMandSSD(§4.3).
neededforcaching. ForFigure1,wereplay350TraceLab To tackle limited SSD parallelism, SARIA combines di-
sessions following the setup described in §5.1. Increasing rectGDSreloadswithlazypartialpromotiontoaccelerate
thecachesizefrom3Mtokenslotsto32Mtokenslots(the subsequentcachereuse.AfterarequestreloadsSSD-hitKV
capacityrequiredforfive-yearendurance)onlyboostscache cacheintoGPUmemorythroughGDSandcompletesprefill,
hitratefrom82.0%to84.8%.Assuch,usinglargerSSDsis lazypromotioncopiessomeSSD-hitblocksfromHBMinto
noteconomicalintermsoftheperformancegain. DRAM.WechooseapromotionratiosothatDRAMandSSD
canfullyutilizetheGPU’singressbandwidthwhileavoiding
Challenge2:LimitedSSDparallelism.ThenumberofSSDs
DRAMcapacitycompetition.Subsequentrequestscanthen
onGPUserversisusuallytoosmalltomatchGPUingress
reloaddifferentportionsoftheKVcachefromDRAMand
bandwidth. For example,AWS’s p5.48xlarge provides 8
SSDconcurrentlytoreduceSSDtraffic(§4.4).
H100GPUsand8localNVMedevices[1],whereasatleast
Loading KV cache concurrently from both DRAM and
4x4PCIeSSDsareneededtomatchonex16GPUlinkatthe
SSDrequirescarefulcoordinationforgoodperformance.We
samePCIegeneration[18]. Addingmoredrivesconsumes
observethatDRAMtransferscanconsumemostoftheGPU
bays and PCIe resources and increases cost. With limited
ingressbandwidthandleavealongSSD-onlycompletiontail.
deviceparallelism,SSDbandwidthremainsabottleneckwhen
reloadingthecachedprefixes.
Assuch,SARIA’scoordinatorpacesDRAMtransferswhile
GDSisactiveandonlyrestoresfullDRAMbandwidthafter
ExistingSSD-basedKVcachesystemsadopttworeload
SSDtransferfinishes.TofullyutilizeGPUingressbandwidth,
paths.ThefirstfetchesSSD-residentKVcachedirectlyinto
SARIAalsousestheresidualGPUingressbandwidthtoserve
GPU memory via GDS [8,19]. Since this path bypasses
DRAMreloadsforotherrequests(§4.5).
DRAM,DRAM copies are not created for SSD hits,a be-
havior we refer to as zero promotion. Repeated SSD hits
WeimplementSARIAonLMCacheMPandevaluateitus-
ingTraceLabandWEKAworkloads,coveringfull-attention
thereforeremainboundbySSDbandwidth.Thesecondpath
andhybridmodelsandsingle-anddual-SSDconfigurations.
loadsSSD-residentKVcacheintoDRAMbeforetransferring
ittotheGPU[28].Thesesystemsretainallreloadedblocks
Theresultsshowthattheselectiveadmission of SARIA re-
ducesSSDwritevolumeby57.1–58.9%comparedwiththe
inDRAMforsubsequentreuse,whichwerefertoasfullpro-
fullduplicatebaseline.ThemeanTTFTisalsoreducedby
motion.AlthoughthesecopiesreducesubsequentSSDreads,
42.3–79.7%overDRAM-onlycaching.Theablationstudy
the promoted KV cache blocks compete with other active
showsthatpartialpromotion,tieroverlap,andcoordination
sessionsforDRAM.
together reduce the cache reload time by 26.7–31.9% for
OurSystemSARIA.Totacklethetwochallenges,wepresent requestsintheslowestquartile.
SARIA(SelectiveAdmissionandReloadforInferenceAccel-
Tosummarize,wemakethefollowingcontributions:
eration),aDRAM–SSDKVcachingsystemforagentwork-
loads.ThecentralideaistoguideSSDpersistenceandpromo- • WeidentifyshortSSDendurancefromeagerpersistence
tionusingsessionreusehistory.SARIA’spersistence-aware andlimitedreloadbandwidthfrominsufficientSSDparal-
SSDadmissionpreservesKVcacheblockswhosereuseex- lelismastwokeychallengesforSSD-basedKVcaching.
tendsbeyondDRAMresidency,whilelazypartialDRAMpro- • WedesignSARIAwithPersistence-AwareSSDAdmission,
motionandcoordinatedmulti-sourcereloadcombineGDS whichusesevictionandreusehistorytoreduceSSDwrites
readswithDRAMreplicastoacceleratesubsequentreloads. whilepreservingcachebeyondDRAMresidency.Tomiti-
Together,theyreduceSSDwritesandreloadlatencyunder gatelimitedSSDparallelism,SARIAcombinesdirectGDS
limitedSSDcapacityandparallelism. reloads with partial DRAM promotion and coordinated
ToaccountforfiniteSSDendurance,SARIAusesrepeated multi-sourcetransferstoreducereloadlatency.
DRAM evictions as evidence that a block needs SSD per- • WeevaluateSARIAacrossfull-attentionandhybridmod-
sistence. This is because requests within the same session elsundervaryingcachepressureandSSDconfigurations,
oftenreuseKVcacheblockswhiletheyremaininDRAM, quantifyingitswrite–performancetrade-offandthebenefits
2
L0 GPU GPU GPU Table1:Per-H200SSDendurancerequirementsundercontin-
uousfullduplicatecaching,assumingafive-year,1-DWPD
L1 DRAM DRAM DRAM SSD rating.The4-TBlifetimeisthetimetoexhausttheratedhost-
writebudget;minimumcapacitysustainsthesamewriterate
retliF
| L2 | SSD | | SSD | | | | forfiveyears. | | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | ------------- | --- | --- | --- | --- | --- | --- | --- |
Min.capacity
| | w/oGDS | | GDS | | | | | | | KV/s | 4-TBSSD | | | |
| --- | ------ | --- | --- | --- | --- | --- | --- | --- | --- | ---- | ------- | --- | --- | --- |
SARIA
| | | | | | | | Model | | (MiB/s) | | lifetime(d) | | for5yr(TB) | |
| ------ | --- | -------- | ------------------- | --- | ----- | ---- | ----------------- | --- | ------- | ----- | ----------- | ---- | ---------- | ----- |
| | | | | | | | Qwen3-30B-A3B[27] | | | 6,515 | | 12.4 | | 590.2 |
| Figure | 2: | KV cache | data pathcomparison | | among | DRAM | | | | | | | | |
| | | | | | | | MiniMax-M2.5[21] | | | 893 | | 90 | | 80.9 |
staging(w/oGDS),GDS,andSARIA.SARIAselectivelyper-
| | | | | | | | Qwen3-235B[30] | | | 383 | | 210 | | 34.7 |
| --- | --- | --- | --- | --- | --- | --- | -------------- | --- | --- | --- | --- | --- | --- | ---- |
sistsKVcachetoSSDandenablesdirectreloadsfromboth
| | | | | | | | GLM-5.2[29] | | | 181 | | 445 | | 16.4 |
| ---- | ------- | --- | --------------------- | --- | ------ | ------- | ----------- | --- | --- | --- | --- | --- | --- | ---- |
| DRAM | andSSD. | | Orange andblackarrows | | denote | offload | | | | | | | | |
andreloadpaths,respectively.
| | | | | | | | host-write | budget,which | | a sustained | | B-GB/s | write | stream |
| --- | --- | --- | --- | --- | --- | --- | ---------- | ------------ | --- | ----------- | --- | ------ | ----- | ------ |
ofitsreloadmechanismsthroughend-to-endexperiments consumesin1,825C/(86.4B)days.
andablations.
FullduplicatecachingpersistseverynewlycomputedKV
block,sowritetrafficscaleswiththecomputedtokenrateand
| | | | | | | | KV bytes | pertoken. | Table | 1 uses | published | | per-H200 | KV |
| --- | --- | --- | --- | --- | --- | --- | -------- | --------- | ----- | ------ | --------- | --- | -------- | --- |
2 BackgroundandChallenges
generationratestoestimatea4-TBSSD’slifetimeandthe
2.1 HierarchicalKVCaching capacityneededtosustainfiveyearsofwrites.Qwen3-235B
| | | | | | | | and GLM-5.2 | use | FP8 | KV; Qwen3-30B-A3B | | | uses | BF16 |
| --- | --- | --- | --- | --- | --- | --- | ----------- | --- | --- | ----------------- | --- | --- | ---- | ---- |
KVcachingallowslaterrequeststorestorecachedprefixes
KVwith4K-tokeninputsandbatchsize32[24,25,40].For
and prefill only unmatched suffixes. CachedAttention [9], MiniMax-M2.5,weusethereportedKVthroughputat32K-
Mooncake[22],andLMCache[3]extendtheKVcachefrom
tokeninputsdirectly.
GPUmemorytohostDRAMandSSD.DRAMprovidesfast
| | | | | | | | At 1 DWPD,sustaining | | | B GB/s | per | GPU | for five | years |
| --- | --- | --- | --- | --- | --- | --- | -------------------- | --- | --- | ------ | --- | --- | -------- | ----- |
reloads,whileSSDextendscapacity.SSDreloadseitherstage
| | | | | | | | requiresatleastC | | =86.4BTBofaggregateSSDcapacity | | | | | |
| ----------------------------------------------- | --- | --- | --- | --- | --- | --- | ---------------- | ------------------------------------------- | ------------------------------ | --- | --- | --- | --- | --- |
| throughDRAMorusedirectI/OsuchasSPIN[2]andGPUDi- | | | | | | | | | min | | | | | |
| | | | | | | | perGPU. | Fullduplicatecachingcanthereforerequiresub- | | | | | | |
rectStorage(GDS)[18].TheGDSpathsinFlexKV[8]and
| | | | | | | | stantially | more SSD | capacity | | for write | endurance | | than for |
| --- | --- | --- | --- | --- | --- | --- | ---------- | -------- | -------- | --- | --------- | --------- | --- | -------- |
DynamoKVBM[19]reloadSSDblockswithoutretaining
effectivecaching.
DRAMcopies(zeropromotion);HiCache’sDRAM-staged
pathsupportsfullpromotion,retainingallreloadedblocksfor
subsequentrequests[28].TuttishowsthatSSDcachingcan
| approachDRAM-backedperformance | | | | | withsufficientSSD | | | | | | | | | |
| ------------------------------ | --- | --- | --- | --- | ----------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
2.3 LimitedSSDReloadBandwidth
parallelismandoptimizedI/O[23].
| | Different | model | architectures | impose | different | require- | | | | | | | | |
| --- | --------- | ----- | ------------- | ------ | --------- | -------- | --- | --- | --- | --- | --- | --- | --- | --- |
ApproachingtheGPU’sPCIeingressbandwidthwithSSD-
| ments | on | KV cache | access | and reuse. | Full-attention | (FA) | | | | | | | | |
| ----- | --- | -------- | ------ | ---------- | -------------- | ---- | --- | --- | --- | --- | --- | --- | --- | --- |
backedKVreloadrequiresmultipledrivesoperatinginpar-
| KVcanbereusedatanymatchedprefixboundary. | | | | | | Hybrid | | | | | | | | |
| ---------------------------------------- | --- | --- | --- | --- | --- | ------ | -------- | ------------ | --- | -------- | --- | ------- | ---------- | --- |
| | | | | | | | allel. A | typical NVMe | | SSD uses | an | x4 PCIe | link,while | a |
modelscombiningFAwithrecurrentlayers,suchaslinear
| | | | | | | | PCIe-attached | GPU | typically | | uses x16. | At | the same | PCIe |
| --------- | --------- | --- | ------- | ------------ | --- | ------------- | ------------- | --- | --------- | --- | --------- | --- | -------- | ---- |
| attention | (LA),also | | require | a checkpoint | at | that boundary | | | | | | | | |
becauselaterstatescannotgenerallyberolledback[11,20]. generation,atleastfourtypicalx4NVMeSSDsmustoperate
inparalleltomatchonex16GPUlink[18].
| Theiraccess | | patterns | therefore | differ: | FA blocks | can serve | | | | | | | | |
| ----------- | --- | -------- | --------- | ------- | --------- | --------- | --- | --- | --- | --- | --- | --- | --- | --- |
prefix hits ending at different positions, while a recurrent DenseGPUserverscommonlyprovidefewerdrivesthan
checkpointisaccessedwhencomputationresumesatitsspe- neededtomatchaggregateGPUingressbandwidth.Acom-
cificboundary. mon configuration in mainstream GPU servers pairs eight
GPUswitheightlocalNVMeSSDs[4].Forexample,AWS’s
2.2 Endurance-ConstrainedSSDProvisioning p5.48xlargehaseightH100GPUsandeightlocalNVMe
| | | | | | | | devices | [1]; GIGABYTE’s | | eight-H200 | | configuration | | sup- |
| --- | --- | --- | --- | --- | --- | --- | ------- | --------------- | --- | ---------- | --- | ------------- | --- | ---- |
Repeatedprogram/erasecycleswearNANDflash,limiting portsatmosteighthot-swapPCIe5.0NVMebays[10],while
SSDwriteendurance.Manufacturersrateenduranceinter- Lenovo’seight-B200configurationsupportsatmostsixteen
abyteswritten(TBW)ordrivewritesperday(DWPD),the NVMe bays [15]. Adding drives consumes bays and PCIe
daily write volume normalized by capacity [16]. We con- resourcesandincreasescost,whilesharedlinkscanlimitag-
siderstandard-enduranceTLCSSDsratedat1DWPDover gregate bandwidth [18]. Long-prefix reloads can therefore
five years [14]. A C-TB SSD therefore has a 1,825C-TB remainbandwidth-bound.
3
0.5
0.4
0.3 0.2
0.1
0.0
0 0.2 0.4 0.6 0.8 1
Per-request DRAM read ratio
tih-ehcac
fo
FDC
(a) Reload DRAM / DISK ratio 1.0
0.8
0.6
0.4
0 4 8 12 16 20 24
DISK reuses per write
setirw
KSID fo
FDC
(b) DISK reuse
400
300 200
100
0
Avg P10 P5
12h (lower load) 4h (medium load) 2h (higher load)
Figure 3: GLM-5.2 reuse under full duplicate caching: (a)
DRAMreadratiosamongcache-hitrequests.(b)subsequent
SSDreadsperwrite.
3 ObservationsandMotivations
Tocharacterizesession-levelKVreuseundercachepressure,
we analyze the same 350 reconstructed TraceLab sessions
usedinourevaluation[31,43],averaging18.4requestsper
session and 83.7K prompt tokens per request. Section 5.1
describes workload construction; Table 2 summarizes the
24-hourreplay.
Toavoiddistortingmulti-turnarrivalpatternsbyscaling
inter-turntimes[35],wepreserveeachsession’srequestse-
quenceandinter-requestintervals,varyingonlysessionstart
times. We place starts within 12-,4-,and2-hourwindows,
denotedlower,medium,andhigherload,respectively.Shorter
windowsmakemoreactivesessioncontextscompeteforthe
sameDRAMcapacity,raisingcachepressurewithoutchang-
ingwithin-sessionreusetiming.
3.1 KVReuseunderCachePressure
Weobservethatrequestswithinanagentsessionoftenreuse
overlappingcontextovershortintervals.Suchtemporallocal-
ityallowsDRAMtoservemuchofthereuseaslongasthe
correspondingKVcacheremainsresident.Wethereforeex-
aminehowcachepressureaffectstheroleofSSDpersistence,
focusing on how much reusable KV cache can already be
servedfromDRAMandhowoftenanSSDcopyisactually
readafterbeingwritten.
Toquantifytheseeffects,wesimulatefullduplicatecaching
underthe three loadsettings. We modela 100 GB DRAM
cache forGLM-5.2 using approximately2.2 million token
slots,basedonitsFP8KVfootprint.SSDcapacityissufficient
toavoideviction,andSSDreadsdonotrepopulateDRAM.
Figure3(a)showsthefractionofcache-hittrafficservedby
DRAM.Inthelower-load12-hoursetting,79.3%ofcache-hit
requestsobtainalltheircachedblocksfromDRAM,which
supplies76.9%oftotalcache-readbytes.Asloadincreases,
moreactivesessionscompeteforthefixedDRAMcapacity,
causingreusableblockstobeevictedearlier.Inthehigher-
load2-hoursetting,thesefractionsfallto52.4%and44.2%,
)sm(
emit
refsnart
.qeR
12h (lower load) 4h (medium load) 2h (higher load)
379 390 350 345
321 310
281 277 206208 226229 214221 244256 231 252
927878 977778 105 7881
Avg P10 P5 Avg P10 P5
Zero promotion Half promotion Full promotion
Figure 4: GLM-5.2 request transfer time under zero, half,
andfullpromotion(lowerisbetter).P10andP5averagethe
slowest10%and5%ofrequests,respectively.Halfpromotion
improvestailtransfertimeoverfullpromotion,withalarger
advantageathigherload.
respectively,substantiallyincreasingrelianceonSSD.
Figure3(b)measuresthevalueofindividualSSDwrites
bycountinghowmanysubsequentSSDreadseachpersisted
blockreceivesthroughtheendofthereplay.Underlowerload,
82.5%ofSSDwritesreceivenosubsequentSSDread.This
fractiondecreasesto65.0%inthe4-hoursettingand44.3%
in the higher-load 2-hoursetting. Thus,underlowercache
pressure,DRAMalreadyservesmuchoftheobservedreuse,
leavingmanyeagerlypersistedSSDcopiesunused.Ascache
pressureincreases,however,reusableblocksaremorelikely
tooutlivetheirDRAMresidency,makingSSDpersistence
increasinglyvaluable.
Observation1.Session-leveltemporallocalityallowsDRAM
to serve most KV reuse under lower cache pressure, leav-
ing many SSD writes unused; as cache pressure increases,
more reuse extends beyond DRAM residency and becomes
dependentonSSDpersistence.
3.2 DRAMCompetitionunderFullPromotion
Thesession-leveltemporallocalityobservedabovealsocre-
atesanopportunityonthereloadpath.AfteranSSD-resident
blockisreloadedthroughGDS,retainingacopyinDRAM
canacceleratesubsequentreusebyreducingrelianceonlim-
itedSSDbandwidth.However,promotedreplicasconsume
additionalDRAMcapacityandcompetewiththeKVcache
ofactivesessions.Wethereforeexaminewhetherpromoting
moreSSD-hitKVcachealwaysreducestransfertimeascache
pressureincreases.
Wecomparezero,half,andfullpromotionunderthethree
loadsettings,holdingtheworkloadandallothercacheset-
tingsfixedwithineachsetting.Transfertimesaremeasured
underidealoverlapbetweenDRAMandSSDtransfers,with
GDSbandwidthsettohalftheGPU-ingressbandwidth.Half
promotion uses a fixed target of 50% DRAM coverage of
thecurrentlyreusedprefix:itcreatesonlyenoughreplicasto
reachthattarget,countingblocksalreadyinDRAM.Figure4
reports mean request transfer time and the mean times for
4
the slowest 10% and 5% of requests,denoted P10 and P5, NewKV DRAM SSD Promoted Control
respectively.
GPU / HBM
| Both half | and full promotion | reduce | mean | transfer | time | | | | | |
| --------- | ------------------ | ------ | ---- | -------- | ---- | --- | --- | --- | --- | --- |
relativetozeropromotion,butfullpromotiondoesnotcon-
noitomorP
| sistentlyimproveonhalfpromotion.Underlowerload,both | | | | | | daolffO | | | | |
| --------------------------------------------------- | --- | --- | --- | --- | --- | ------- | ------ | ------ | --- | --- |
| | | | | | | | )4.4§( | daoleR | | |
achieveameanofapproximately78ms,withsimilarP5times Coordinator
| of226msforhalfpromotionand229msforfullpromotion. | | | | | | | | | | (§4.5) |
| ------------------------------------------------ | ----------------- | --- | ----------- | --- | ----- | ---- | --- | -------- | ---- | ------ |
| The tailgapwidens | as loadincreases: | | undermedium | | load, | | | | | |
| | | | | | | DRAM | | Prewrite | SSDs | |
halfpromotionreducesP5from256to244ms;underhigher
| load,it reduces | P10 from | 252 to 231 | ms | and P5 | from 310 | | | | | |
| --------------- | -------- | ---------- | --- | ------ | -------- | --- | --- | --- | --- | --- |
| | | | | | | MRU | | LRU | MRU | LRU |
to277ms,whilealsoloweringthemeanfrom81to78ms.
Thus,fullpromotionretainsmuchoftheaveragebenefitbut
Admit
| delivers smallertailimprovements | | | athigherloadthan | | half | | | | | |
| -------------------------------- | --- | --- | ---------------- | --- | ---- | --- | --- | --- | --- | --- |
Evict
| promotion. | | | | | | | | | | Filter |
| ---------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ------ |
TheadditionalreplicasalsoincreaseDRAMreplacement.
MetadataandPlanning
Inthe2-hourreplay,cumulativeDRAMblockevictionsrise
SSD Admission (§4.2)
from 59,589 with zero promotion to 97,097 with half pro- (§4.3)
| motion and | 175,314 with | full promotion. | | More aggressive | | | | | | |
| ---------- | ------------ | --------------- | --- | --------------- | --- | --- | --- | --- | --- | --- |
promotion therefore creates more replicas but also evicts Figure5:OverviewofSARIAarchitectureanddatapaths.
moreresidentKVcacheblocks,potentiallyremovingreusable
| blocks before | theirnextaccess. | Togetherwiththe | | | transfer- | | | | | |
| ------------- | ---------------- | --------------- | --- | --- | --------- | --- | --- | --- | --- | --- |
time results,these counts indicate that limiting promotion currentlocationofeachKVblock.Duringlookup,thecache
| | | | | | | manageruses | this metadata | to construct | | an ordered reload |
| --- | --- | --- | --- | --- | --- | ----------- | ------------- | ------------ | --- | ----------------- |
canalleviateDRAMcompetitionandimprovetailtransfer
performanceundercachepressure. plan for the reusable prefix,which may span both DRAM
Observation2.Session-leveltemporallocalitymakesDRAM andSSD(§4.3).Afterprefillcompletes,LazyPartialDRAM
promotion beneficial, but aggressive promotion intensifies PromotioncopiesselectedSSD-hitblocksfrom HBMinto
DRAMcompetitionundercachepressure.Reducingthepro- DRAM to accelerate subsequent reuse (§4.4). Finally, the
transfercoordinatoruseseachrequest’sreloadplantopace
motionratiocanalleviatethiscompetitionandimprovetail
transferlatency. DRAMreadsalongsideGDStransfers,allowingthetwopaths
toefficientlyshareGPUingressbandwidth(§4.5).
4 SARIA
4.2 Persistence-AwareSSDAdmission
4.1 SystemOverview
EverySSDwriteconsumesafiniteendurancebudget,even
SARIA managesahierarchicalDRAM–SSDKVcachefor if the persisted KV cache is never read from SSD (§2.2).
servingagentworkloadsunderlimitedSSDenduranceand Observation1(§3.1)showsthatDRAMsuppliesmostcached
reloadbandwidth.ItselectivelypersistsKVcacheusingreuse reuseundernormalloadandmanyeagerlypersistedcopies
andevictionhistoryaccumulatedinDRAM,andmaintains providenoreadbenefit:inthereferencereplay,82.5%ofSSD
| | | | | | | writes under | full duplicate | caching | receive | no subsequent |
| --- | --- | --- | --- | --- | --- | ------------ | -------------- | ------- | ------- | ------------- |
contextmetadatatoapplyadmissionconsistentlyacrosshy-
bridKVtypesandrecoverprefixesdistributedacrossstorage SSDread.ThismotivatesanadmissionfilterthatdefersSSD
tiers. When a request reloads SSD-backed KV cache,lazy persistencewhileDRAMservesreuseandseeksevidencethat
partialpromotioncreatesselectedDRAMreplicasforsubse- retainingtheKVcachebeyonditsDRAMresidencywould
| quentreuse,withthepromotionvolumejointlydetermined | | | | | | bebeneficial. | | | | |
| -------------------------------------------------- | --- | --- | --- | --- | --- | ------------- | --- | --- | --- | --- |
bythereuserecencyofDRAMevictioncandidatesandthe
Anaturaladmissionsignalisrequestpopularity.HiCache,
DRAM-to-SSDbandwidthratio.Acoordinatorschedulesthe for example, uses access counts to filter HBM-to-DRAM
resultingDRAMandGDStransferstoefficientlyshareGPU backups[28],butdoesnotmakeaseparateadmissiondecision
ingressbandwidth. forSSDendurance.Forpredominantlysession-privateagent
Figure5illustrateshowSARIAmanagesKVcachethrough- context,however,thesecountslargelydescribereusewithina
outarequest’slifecycle.NewlygeneratedKVcacheblocks sessionanddonotreflecttheDRAMcompetitionfromother
arefirstplacedinDRAM.Persistence-AwareSSDAdmission sessions. Increased competition from otheractive sessions
thenusesaccumulatedreuseandevictionhistorytodetermine canshortenablock’sDRAMresidencyevenwhenitsown
whichblocks shouldbe persistedto SSD forreuse beyond accesspatternremainsunchanged.Moreover,acumulative
theirDRAMresidency(§4.2).Unifiedcontextmetadatapre- accesscountdoesnotdistinguishreuseconcentratedwithin
servesthishistoryacrossDRAMresidenciesandtracksthe ashortburstfromreusespreadoverlongerintervals.Blocks
5
| | | | DISK reuses per block | | | | | SSD write | DRAM | SSD | Promoted | | DRAM + SSD |
| --- | --- | --- | --------------------- | --- | --- | --- | --- | --------- | ---- | --- | -------- | --- | ---------- |
0 1–3 ≥4 (a) SSD admission (§4.2) (b) Context metadata (§4.3)
| | (a) Lower load (12h) | | | (b) Higher load (2h) | | | | Full duplicate | | | | | |
| --- | -------------------- | --- | --- | -------------------- | --- | --- | --- | -------------- | --- | --- | --- | --- | --- |
skcolb desuer fo rebmuN 5 writes Block placement Context radix tree
| | 24.4k | | | 26.1k | | | | DRAM A B | C D E | DRAM | | | A |
| --- | ----- | --- | --- | ----- | --- | --- | --- | -------- | ----- | ---- | --- | --- | --- |
24k
| | | | | | | | | | | A F | G B | | |
| --- | --- | ----------- | --- | --- | ----- | --- | --- | ------- | ----- | --- | --- | --- | ---- |
| | | | | | | | | SSD A B | C D E | | | | B G |
| | | | | | 16.3k | | | | | | | | Req3 |
| | 16k | 15.0k 14.5k | | | | | | | | | | C | E |
12.5k
| | | | | | | | | Admission Filter | 2 writes | SSD | | | |
| --- | --- | --- | ---- | --- | --- | ---- | --- | ---------------- | -------- | --- | ----- | ----- | ----- |
| | | | 7.2k | | | | | | | | | D | F |
| | 8k | | | | | 5.5k | | DRAM A B | C D E | A B | C D E | | |
| | | | | | | | | | | | | Re q1 | Re q2 |
1.2k
| | 0 | | | | | 291 | | | Filter | Req reload plan | | | |
| --- | ------------------- | --------------- | --- | ------------------- | --------------- | --- | ------------------ | ----- | --------------------- | --------------- | ----- | --- | ----- |
| | 1–7 | 8–15 16–3132–63 | ≥64 | 1–7 | 8–15 16–3132–63 | ≥64 | | | | | | | |
| | | | | | | | | SSD A | D | | | | |
| | Request reuse count | | | Request reuse count | | | | | | Req1 | Req2 | | Req3 |
| | | | | | | | AA: Popular prefix | | DD: Repeated eviction | A B | C D A | B E | F A G |
| | | | | | | | Reuse ≥ threshold | | Evictions ≥ threshold | | | | |
Figure6:GLM-5.2SSDreusebyrequestpopularityunder
12-hourand2-hoursession-startwindows. Figure7: (a)Persistence-awareSSDadmission.(b)Unified
metadataandmixed-sourcereloadplanning.
withsimilarpopularitycanthereforehaveverydifferentneeds
forSSDpersistence.
asitremainsonSSD.
| Figure | 6 | illustrates | this mismatch | | by grouping | reused | | | | | | | |
| ------ | --- | ----------- | ------------- | --- | ----------- | ------ | --- | --------- | ---------- | ------- | ---------- | --- | ------------ |
| | | | | | | | | Admission | determines | whether | to persist | a | block; write |
blocksfromthe12-hourand2-hoursettingsofObservation1 schedulingdetermineswhenpersistencebegins.Whenaneli-
accordingtotheirrequest-reusecounts.Eachbarseparates
gible,non-persistedblockreachestheDRAMevictionpath,
| blocks | with | zero, | one to three, | or at least | four | SSD reads. | | | | | | | |
| ------ | ---- | ----- | ------------- | ----------- | ---- | ---------- | --- | --- | --- | --- | --- | --- | --- |
SARIAwritesittoSSDbeforereclaimingitsDRAMstorage.
Amongblocksreused16–31times,thefractionreceivingat
TheDRAMcopyremainsavailableasthewritesourceun-
leastoneSSDreadincreasesfrom29.0%underlowerloadto
tilthetransfercompletes.Persistencecanalsobeginearlier.
60.2%underhigherload.Evenamongblocksreusedatleast ThereusepathcaninitiateanasynchronouswritewhenR
b
64times,44.5%receivenoSSDreadinthe12-hoursetting.
| | | | | | | | reachesθ | R .PrewriteevaluatescandidatesusingE | | | | | b +1,allow- |
| --- | --- | --- | --- | --- | --- | --- | -------- | ------------------------------------ | --- | --- | --- | --- | ----------- |
Requestpopularityalonethereforedoesnotcapturetheneed
| | | | | | | | ing | blocks thatwouldqualify | | attheirnexteviction | | | to start |
| --- | --- | --- | --- | --- | --- | --- | --- | ----------------------- | --- | ------------------- | --- | --- | -------- |
forSSDpersistenceundercachecompetition.
writingwhilestillresidentinDRAM.Whensufficientlead
SARIAusesrepeatedDRAMevictionsasapracticalsignal
timeandwritebandwidthareavailable,theseasynchronous
ofpersistencedemand.AKVblock’sfirstevictionindicates
writesmoveSSDtrafficofftheDRAMreclamationcritical
thatDRAMcannolongerretainitunderthecurrentwork-
path.
load.Iftheblockissubsequentlyreusedandlaterreachesan-
otherevictionboundary,itshistoryincludesreusethatDRAM
4.3 UnifiedMetadataandReloadPlanning
couldnotcover.Thiscross-residencyhistoryincorporatesthe
competitionexperiencedbythatblockwithoutexplicitlyesti-
| | | | | | | | Applying | admission | | independently | to physical | | blocks can |
| --- | --- | --- | --- | --- | --- | --- | -------- | --------- | --- | ------------- | ----------- | --- | ---------- |
matingitsreuseintervalorDRAMretentiontime.Repeated
leaveahybridmodelwithouttheKVcacheneededtorecover
evictiondoesnotguaranteeanotherreturn,butprovidesdirect
acontextprefix.Full-attention(FA)KVthroughapositionc
| evidence | thatSSD | | persistence | couldhave | protecteduseful | | | | | | | | |
| -------- | ------- | --- | ----------- | --------- | --------------- | --- | --- | --- | --- | --- | --- | --- | --- |
remainsusefulwhenlaterrequestsextendbeyondc,whereas
KVcacheacrossanearliereviction.Weadditionallyretaina
alinear-attention(LA)recurrentstateatcisreusedasacheck-
request-reusethresholdsothatexceptionallypopularblocks
pointatthatparticularboundary[11,20].Thesecomponents
canqualifyforpersistencebeforeexperiencingrepeatedevic-
canthereforeaccumulatedifferentaccessandevictionhisto-
tions.
riesdespiterepresentingthesamecontextposition.Indepen-
Foreachlogicalcontextblockb,SARIAtracksitsDRAM-
dentadmissionmaythereforepersistFAblockswithoutthe
| evictioncountE | | andrequest-reusecountR | | | | acrossDRAM | | | | | | | |
| -------------- | --- | ---------------------- | --- | --- | --- | ---------- | ---------------------------------------------------- | --- | --- | --- | --- | --- | --- |
| | | b | | | b | | LAcheckpointneededtoreusethatprefix.Selectivepersis- | | | | | | |
residencies.Theevictioncounteradvanceswhentheblock
tenceandindependentreplacementalsodistributereusable
isselectedataDRAMevictionboundary.Ablockbecomes
blocksacrossDRAMandSSD,producinginterleavedsources
eligibleforSSDpersistencewhen
suchasDRAM–SSD–DRAM.Recoverymustthereforees-
admit(b)=[E ≥θ ]∨[R ≥θ ]. (1) tablishavalidprefixboundaryandassignasourcetoevery
| | | | b | E | b R | | | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
cacheobjectrequiredatthatboundary.
The two thresholds expose a trade-off between SSD write SARIAorganizescontextmetadatainaradixtree,follow-
trafficandpersistencecoverage.Largerthresholdsfiltermore ingtheprefixorganizationusedinHiCacheandUnifiedRadix
writes but delay protection,potentially requiring recompu- Cache[11,28].Eachnoderepresentsablock-alignedcontext
tationonearlyreturns;smallerthresholdsadmitKVcache prefixandassociatesitsphysicalcacheobjectswithshared
blockssooneratgreaterwritecost.Oncepersisted,ablock reuse and eviction counts and per-object storage locations.
canservesubsequentreuseafterDRAMevictionforaslong Admissionisevaluatedatthiscontextgranularityandapplied
6
NewKV DRAM SSD Promoted reloadpaths.SinceDRAMandGDScanservedifferentpor-
tionsofaprefixconcurrently,DRAMonlyneedstosupply
| GPU | GPU | GPU | | | | | | | | | |
| --- | --- | --- | --- | ------------- | --- | ------------ | --- | ------ | -------- | --- | ------ |
| | | | | the bandwidth | | not provided | by | SSD to | saturate | the | shared |
GPUingressbandwidth.
| DRAM | SSD DRAM | SSD DRAM | SSD | | | | | | | | |
| ---- | -------- | -------- | --- | ---- | ---------------------------------------- | --- | --- | --- | --- | --- | --- |
| | | | | LetV | denotethetotalbytesofasubsequentreload,B | | | | | | the |
disk
| | | | | sustainedGDS-to-GPUbandwidth,andB | | | | | GPU thesharedGPU- | | |
| ------------- | ------------ | --------------- | --- | --------------------------------- | --- | --- | ---- | --- | ----------------- | ---------- | --- |
| | | | | ingressbandwidth,whereB | | | | <B | .AssumingDRAM | | |
| | | | | | | | disk | GPU | | | |
| | | | | cansustaintheresidualbandwidthB | | | | GPU | −B disk | whileGDSis | |
| (1)Reloadfrom | (2)Promoteto | (3)Multi-source | | | | | | | | | |
SSD DRAM reload active,thetargetfractionofreloadbytesservedfromDRAM
is
B
Figure8: IllustrationoftheLazyPartialDRAMPromotion r⋆=1− disk (2)
B GPU
workflow.
| | | | | Servingr⋆·V | bytesfromDRAMand(1−r⋆)·V | | | | | bytesfrom | |
| --- | --- | --- | --- | ----------- | ------------------------ | --- | --- | --- | --- | --------- | --- |
SSDallowsthetwopathstofinishtogetherwhilefullyutiliz-
totheassociatedobjects.Thisallowsacheckpointtoshare ingGPUingress,yieldingareloadtimeofV/B .Promot-
GPU
theadmissionevidenceaccumulatedbyitscontextinsteadof
ingbeyondthisfractionconsumesadditionalDRAMcapacity
requiringeverycomponenttoindependentlyreachthethresh- withoutfurtherreducingreloadtime.
old. Metadataisretainedacrossevictionssothatreturning This bandwidth balance guides the target DRAM share
blockspreservetheirhistoryacrossresidencies.Wemanage r∈[0,1],whichalsogivestheblockfractionforequal-sized
metadatawithaslidingtimewindow,reclaimingablock’s blocks.IfarequestcurrentlyreloadsD blocksfromDRAM
0
metadataonlyafteranentirewindowpasseswithoutreuseor
| | | | | and S 0 blocks | | from SSD, | the | number | of additional | | SSD- |
| -------------- | ----------------- | -------- | ------------ | ---------------------------------- | --- | ---------- | --- | ------ | ------------- | --- | ---- |
| recomputation. | | | | backedblockstargetedforpromotionis | | | | | | | |
| At lookup, | the cache manager | combines | the KV cache | | | | | | | | |
| | | | | | P | =max(0,r(D | | +S | )−D ) | | (3) |
availableinDRAMandSSDintoaunifiedreloadplan(Fig- target 0 0 0
ure7(b)).Theplanidentifiesthereusableprefixandselects
| | | | | Blocks already | | resident | in DRAM | count | toward | | the tar- |
| --- | --- | --- | --- | -------------- | --- | -------- | ------- | ----- | ------ | --- | -------- |
eachrequiredobjectfromanavailableDRAMorSSDsource, get, so no promotion is needed when the existing DRAM
preservingblockorderevenwhenhitsalternatebetweentiers. sharealreadyreachesr.Theactualpromotionvolumeisfur-
therboundedbytheeviction-candidatefilterabove,allowing
SARIAtobalancereloadbandwidthagainstcurrentDRAM
4.4 LazyPartialDRAMPromotion
contention.
LimitedSSDparallelismconstrainsGDSreloadbandwidth
(§2.3). Observation2(§3.2)showsthatpromotionreduces 4.5 CoordinatedMulti-SourceReload
meantransfertimebutincreasescompetitionforDRAMca-
pacity. Limitingthepromotionratiocaneasethiscompeti- Thesourcebalancein §4.4assumesthatDRAMandGDS
| | | | | make progress | | concurrently. | However,we | | find | that | simply |
| --- | --- | --- | --- | ------------- | --- | ------------- | ---------- | --- | ---- | ---- | ------ |
tionandreducetailtransferlatency.SARIAthereforecreates
DRAMreplicasondemandwhilelimitingtheirvolume.Af- enabling overlapdoes notensure concurrentprogress. Fig-
ure16showsthatSSDtrafficremainslowduringtheinitial
terarequestreloadsKVcachefromSSDthroughGDSand
completesprefill,SARIAcopiesselectedSSD-hitblocksfrom DRAMburstandreachesitssustainedrateonlyafterDRAM
HBMintoDRAM(Figure8).Ifthesereplicasremainresident, trafficsubsides.Adisk-onlycompletiontailthereforepersists
despiteconcurrentsubmission,leavingGPUingressunderuti-
subsequentrequestscanreusethemfromDRAMinsteadof
| repeatedlyrelyingonthebandwidth-limitedSSDpath. | | | | lized. | | | | | | | |
| ----------------------------------------------- | --- | --- | --- | ------ | --------- | ----------- | --- | ------- | ------- | --- | ------ |
| | | | | SARIA | therefore | coordinates | | the two | sources | by | pacing |
SARIAlimitstheDRAMcompetitionintroducedbypromo-
tionalongtwodimensions.First,itavoidsdisplacingrecently DRAM traffic while GDS work is active. The coordinator
reusedKVcacheblocksinDRAMmerelytoaccommodate consumes each request’s reload plan from §4.3, tracks its
| | | | | remaining | DRAM | and | SSD work, | and | shares | a per-GPU | |
| --- | --- | --- | --- | --------- | ---- | --- | --------- | --- | ------ | --------- | --- |
promotedreplicas.Acandidateblockbmaybeevictedfor
promotion only if t −t reuse ≥P , where t reuse is its last ingressbudgetacrossconcurrenttraffic.LetB DRAM denote
| | now b | τ | b | | | | | | | | |
| --- | ----- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
thestandaloneDRAMreloadbandwidth.WhileSSDworkis
| reusetimeandP | τ isarecencythreshold.Recentlyreusedcan- | | | | | | | | | | |
| ------------- | ---------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
didatesindicatestrongcompetitionforDRAMcapacityand runnable,SARIAlimitstheaggregateDRAMratetoapproxi-
| areprotectedfrompromotion-inducedeviction.SARIAthere- | | | | mately | | | | | | | |
| ----------------------------------------------------- | --- | --- | --- | ------ | ----- | ------ | ---- | ------ | ------- | --- | --- |
| forepromotesonlyasmanyblocksascanbeaccommodated | | | | | Bpace | | | | | | |
| | | | | | | =min(B | DRAM | ,B GPU | −B disk | ), | (4) |
| bycandidatessatisfyingthisthreshold.Whenfewcandidates | | | | | DRAM | | | | | | |
qualify,thepromotionvolumeisreduced,potentiallytozero. reserving sufficient ingress bandwidth for GDS to sustain
Second, SARIA bounds the target promotion volume ac- B .Thecoordinatorenforcesthisratebyboundingindivid-
disk
cording to the relative bandwidth of the DRAM and SSD ualDRAMsubmissionsandspacingthemaccordingtothe
7
Table 2: Selected workloads after filtering and truncation.
Minimumprefilltokensdenotethetotalnumberofpromptto-
kensthatstillrequirecomputationwithanunlimited-capacity
cache.
Totalprompt Minimumprefill
Dataset Sessions Requests tokens(106) tokens(106)
ReqA ReqB ReqC
TraceLab 350 6,431 538.26 36.55
WEKA 170 5,995 930.25 26.43
Workloads.Weevaluateprefillservingusingcoding-agent
ReqA ReqA+B ReqB ReqC
workloads from TraceLab traces of Claude Code and
Codex[31],andSemiAnalysisWekaTrace(WEKA)tracesof
Figure9: Illustrationofmulti-sourcereloadwithandwithout
ClaudeCode[26].Neitherdatasetincludespromptcontents,
transfercoordination.
sowereconstructdeterministicsyntheticinputspreserving
recordedreuse.TraceLabrecordsrequesttokencountsand
available budget. When no SSD workis runnable,DRAM timestamps;itsprefixesarereconstructedfromwithin-session
returnstoitsunthrottledpath. requestlengths.WEKAinputsuserecorded64-tokenblock
Figure9illustratesthecoordinatedschedule.OnceRequest identitiesscopedbysourcetraceandmodel.
AfinishesitsDRAMportion,residualingressadvancesthe We select TraceLab sessions by start time and WEKA
DRAM-onlyRequestBwhileAcontinuesreadingfromSSD. streamsbydatasetorder.EachWEKAstreamformsaninde-
WhenA’sSSDtransferfinishes,BusesthefullDRAMrate. pendentagentsession,includingrequestsafterprefixresets.
RequestC overlaps its DRAM andSSD transfers; afterits Bothworkloadsusea450K-tokencontextlimitand24-hour
SSDportioncompletes,itsremainingDRAMworkalsoruns truncation(Table2),withmeanwithin-sessionrequestinter-
atfullrate. valsof112.1and111.1s,respectively.
The coordinator is configured using estimated DRAM, We apply the replay method from §3 to both datasets,
GDS,andGPU-ingressbandwidths.Inaccuratesettingscan usingadefault12-hoursession-startwindowwithGamma-
causebandwidthunderutilizationortransfercontention,de- distributedgaps(k=1)following[35].
gradingreloadperformance.WhenGDSalonesaturatesGPU Replaypermitsatmostoneactiverequestpersession.Suc-
ingressbandwidth,coordinationoffersnobenefitanditsover- cessorsfollowrecordedidleintervals.Weallowfourrequests
headmayreduceperformance. inflightbydefault;thecumulativeablationandpromotion
sweepuseone.Requestsstopafterthefirstoutputtokentoiso-
lateprefillserving.Thelogicalreplayclockadvancesat6,500
5 Evaluation
computedprefilltokens/s;reportedlatenciesusewall-clock
time.
5.1 ExperimentalSetup
Baselinesandcacheconfiguration.WeimplementSARIA
Models and testbeds. Our primary testbed has one RTX on LMCache MP and integrate Phoenix as its GDS back-
PRO 6000 Blackwell Server Edition GPU with 96 GB of end.Ourimplementationincludesthreeoptimizationstothe
GDDR7memory,twoAMDEPYC9534CPUs,256GBof LMCacheMPdatapath.First,GDScontroloperationsare
hostmemory,andtwo1.92-TBPCIe5.0NVMeSSDs.We asynchronously pipelined to reduce control-path overhead.
evaluate releasedBF16 checkpoints ofQwen3-8B with36 Second,whenDRAMallocationfailsduringHBMoffload,
full-attention layers and Qwen3.5-9B with 8 full-attention thecacheevictsexistingentriesandretriestheallocation,al-
and24gated-delta-network(GDN)layers. lowingoffloadtocontinueundermemorypressure.Third,fol-
Admission and metadata experiments use a secondary lowingTutti[23],weseparateSSDreadandwriteexecution
testbed with a 24-GB A30 GPU and one PCIe 5.0 NVMe toreduceI/Ointerferenceandimproveaggregatethroughput.
SSD. To accommodate its HBM while preserving the at- Two LMCache-based baselines use optimized Phoenix
tention structures, we construct Qwen3-1.7B with 20 full- GDSbackends[3]:GDS-onlystoresKVexclusivelyonSSD;
attentionlayersandQwen3.5-5.6Bwith5full-attentionand LMCache+ uses full duplicate DRAM–SSD caching with
15GDNlayers. Thesenamesdenoteourscaled-downcon- per-object metadata, serial tier loading, and no promotion.
figurations rather than released checkpoints; both execute WealsorunSARIAasDRAM-only(SSDdisabled)andFull
realforwardpasseswithrandomlyinitializedBF16weights, Duplicate(selectiveadmissiondisabled,metadataandreload
following[22,38,41]. mechanismsretained).
8
DO: DRAM-only E5-E2: SARIA (admission θE=(5,4,3,2)) FD: SARIA (full duplicate) LMC+: LMCache+ GO: GDS-only
TTFT (s) CHAR P25 Bad Avg Total disk write / capacity Total disk read / capacity
8×
25.3 Avg P90 threshold 100% 98.2%97.1%97.2% Write budget 532%540%533% 64× 52×
| B9-5.3newQ | | | | | | 82.4% | | | | | | | |
| ---------- | --- | ------- | ----- | ----------- | ---- | ----- | ----- | --- | ---- | --- | --- | --- | --- |
| baLecarT | 16 | 9.2 | | | | 72.5% | 4 × | | | | | | 20× |
| | | 7.0 | | | | 63.5% | | | 219% | 16× | | | 15× |
| | 8.3 | | 5.1 | | | | 2 × | | | | | 8× | |
| | 4 | | | | 50% | | | | 113% | | | | |
| | | | | 1 .8 | | | 1 × | | | 4 × | | 4× | |
| | | 2.8 2.2 | 1 .5 | 1 .6 | | | | | | | | | |
| | 1 | | 1.7 | | | | 0.5 × | 49% | | | 1× | | |
| | | | 1 . 1 | 1 . 2 1 . 2 | 0.0% | | | 0% | | | 0× | | |
| | | | | | 0% | | | | | 1 × | | | |
DO E5 E3 E2 FD LMC+ GO DO E5 E3 E2 FD LMC+ GO DO E5 E3 E2 FD LMC+ GO DO E5 E3 E2 FD LMC+ GO
18.7 Avg P90 threshold 100% 98.6%97.9%96.9% 4× Write budget 347%349%348% 64× 50×
| B8-3newQ | 16 | 13.4 | | | | 84.9% | | | | | | | |
| -------- | --- | ------- | --- | --- | --------------- | ----- | ---- | --- | ---- | --- | --- | --- | --- |
| baLecarT | | 10.2 | | | 65.9%70.5%76.1% | | 2× | | | | | | 16× |
| | | | | | | | | | 149% | 16× | | | 12× |
| | | | 5.9 | | | | | | | | | 6× | |
| | 6.0 | | | 4.4 | 50% | | 1× | | | | | | |
| | 4 | 5.0 4.3 | 2.7 | 2.8 | | | | | 68% | 4× | | 2× | |
| | | | 3.5 | 3.3 | | | 0.5× | | | | | | |
| | | | 2.4 | 2.5 | | | | 39% | | | 1× | | |
| | | | | | | | | 0% | | 1× | 0× | | |
DO E4 E3 E2 FD LMC+ GO 0% DO E4 E3 E2 FD LMC+ GO DO E4 E3 E2 FD LMC+ GO DO E4 E3 E2 FD LMC+ GO
130.5
Avg P90 threshold 100% 87.2%91.4% 97.8%96.1%94.2% Write budget 452%465%417% 12 8 × 91×
| B9-5.3newQ | 64 | | | | | 79.6% | 4× | | | | | | |
| ---------- | --- | --- | --- | --- | --- | ----- | --- | --- | --- | --- | --- | --- | --- |
6 4 ×
| akeW | 39.5 | | | | | | | | | | | | 32× |
| ---- | ---- | --- | ------- | ------- | ------- | --- | ---- | --- | ---- | ----- | --- | --- | --- |
| | 16 | 8.4 | | | | | 2 × | | | 3 2 × | | | 26× |
| | | 4.7 | | | 50% | | | | 138% | | | 18× | |
| | 4 | | 3.2 1.9 | 2.1 2.3 | | | 1 × | | 99% | 1 6 × | | 13× | |
| | | 3.0 | | | | | | 69% | | | 9× | | |
| | 1 | 2.0 | 1.7 | | | | | | | 8× | | | |
| | | | 1.1 | 1.3 1.4 | 0% 0.0% | | 0.5× | 0% | | | 0× | | |
DO E5 E3 E2 FD LMC+ GO DO E5 E3 E2 FD LMC+ GO DO E5 E3 E2 FD LMC+ GO DO E5 E3 E2 FD LMC+ GO
Figure10:End-to-endlatency,cachecoverage,andSSDtrafficonTraceLabandWEKA.TTFTbarsshowmeansandblacklines
showP90.SSDI/OisnormalizedbySSDcachecapacity.Thepalebandinthethirdcolumnspansthe12–24-hourwrite-budget
limits(0.5–1.0×capacity)at1DWPD;writesbelow0.5×meetbothlimits.
5.2 MainResults
Unlessotherwisespecified,DRAM-containingconfigura-
tionsreceivebudgetsof1MtokensforDRAMand10Mto-
kensforSSD.Hybridbudgetsusefull-attentiontokenequiva- End-to-EndPerformance.Figure10evaluateshowSARIA
lents;GDNcheckpointsstillconsumephysicalcapacityand balanceslatencyagainstSSDwritebudgetsandreadband-
contributetoI/O.Dual-SSDconfigurationsstripeeachobject
| | | | | | | | width. | Using | a standard-endurance | | TLC | SSD | rated at 1 |
| --- | --- | --- | --- | --- | --- | --- | ------ | ----- | -------------------- | --- | --- | --- | ---------- |
acrossbothdrives. DWPD[14],weallocateenduranceinproportiontothecon-
WeuseeagerexecutionwithHBMprefixcachingdisabled. figuredcachecapacityC.The12-hoursession-startwindow
Externalcacheblockscontain512tokensfordensemodels and24-hourtracetruncationgivewritebudgetsof0.5Cand
and1,024forhybridmodels;GDNcheckpointsareretained C,respectively,markedbytheshadedband.
every16,384tokens.Bydefault,SARIAusespromotionratio
| | | | | | | | OnWEKAwithQwen3.5-9B,anevictionthresholdθ | | | | | | =3 |
| --- | --- | --- | --- | --- | --- | --- | ----------------------------------------- | --- | --- | --- | --- | --- | --- |
E
r=0.4andshared-ingress/GDS/DRAMbandwidthestimates (E3)stayswithintheone-daywritebudget,reducingwritesby
of48/20/48GB/s(24/10/24GB/sonA30).Thepromotion
78.2%relativetoFullDuplicate(FD)ata78.4%increasein
recency thresholdis fixedatP τ =60 s on the replay clock. meanTTFT.ItstillreducesmeanTTFTbyabout94.9%rela-
Experiment-specificoverridesaccompanytheresults.
tivetoDRAM-only.E2furtherimproveslatencybutexceeds
Metrics.TTFTmeasureswall-clocktimefromsubmission theone-daywritelimit.
tovLLMthroughthefirstoutputtoken;cache-transfertime TraceLabexhibitsthesametrade-offbutrequiresdifferent
spansthefirstexternal-cachelookupthroughvLLM-observed thresholds. For Qwen3.5-9B, E3 reduces writes by 78.7%
relativetoFDbutexceedstheone-daybudgetby13.3%.Its
| load | completion,excluding | | | prefill | computation. | We report | | | | | | | |
| ---- | -------------------- | --- | --- | ------- | ------------ | --------- | --- | --- | --- | --- | --- | --- | --- |
meanand90th-percentile(P90)TTFTinthemaincomparison. meanandP90TTFTare73.7%and72.2%lowerthanDRAM-
LatencyP25badaveragesaremeansofthelargest25%of only.RaisingthethresholdtoE5meetseventhe12-hourwrite
finitelatencysamples. limitwhileretainingmeanandP90TTFTreductionsof66.5%
Cachehitachievementrate(CHAR)istheratioofactual and63.7%.ForQwen3-8B,E3meetstheone-daybudgetwith
80.3%fewerwritesthanFD,whilereducingmeanandP90
cache-hitblockoccurrencestothoseachievablewithunlim-
ited capacity; 100% means all available reuse is captured. TTFT by 27.8% and 45.3% relative to DRAM-only. The
CHARP25badavgrecomputesthisratiooverthecombined thresholdneededtomeetawritebudgetthusdependsonboth
blockcountsofthequarterofeligiblerequestswiththelow- workloadandmodel.
est per-request CHAR. Normalized SSD reads and writes DRAM-assistedreloadalsoreducesSSDreadtraffic:FD
divide cumulative completed payload bytes by configured issues27.2%fewerreadbytesthanLMCache+forbothmod-
SSDcachecapacity,includingGDNstateandtransfersacross elsonTraceLaband18.1%fewerforQwen3.5-9BonWEKA.
bothdrives. Together,admissionandreloadmanagementbalancelatency,
9
| | | | | SARIA (θE=2) | SARIA (θR=4) | | Full Duplicate | | | | | |
| --- | --- | --- | --- | ------------ | ------------ | --- | -------------- | --- | --- | --- | --- | --- |
TTFT Avg (s) CHAR P25 Bad Avg Total disk write / capacity Disk read / cache size
| | | | 100% | | 220% | | | | | | | |
| ------------ | --- | --- | ---- | --- | ---- | --- | --- | --- | --- | --- | --- | --- |
| B9-5.3newQ 3 | | | | | | | | | | 8x | | |
75%
180%
2
| | | | 50% | | 140% | | | | | 6x | | |
| --- | --- | --- | --- | --- | ---- | --- | --- | --- | --- | --- | --- | --- |
| 1 | | | 25% | | | | | | | | | |
100%
4x
| 0 | | | 0% | | | | | | | | | |
| ---------- | --- | --- | ----- | ----- | ---- | --- | --- | --- | --- | --- | ----- | --- |
| 10 | 20 | 30 | 40 10 | 20 30 | 40 | 10 | 20 | 30 | 40 | 10 | 20 30 | 40 |
| | | | 100% | | | | | | | 8x | | |
| B8-3newQ 3 | | | | | 100% | | | | | | | |
75%
| 2 | | | | | | | | | | 6x | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| | | | 50% | | | 80% | | | | | | |
| 1 | | | 25% | | | | | | | | | |
4x
60%
| 0 | | | 0% | | | | | | | | | |
| --- | --- | --- | ----- | ----- | --- | --- | --- | --- | --- | --- | ----- | --- |
| 10 | 20 | 30 | 40 10 | 20 30 | 40 | 10 | 20 | 30 | 40 | 10 | 20 30 | 40 |
Session-start interval (min) (Higher load → Lower load)
Figure11: Cachebehaviorundervaryingservingpressure.Afixedsetof100sessionsisreplayedwithsession-startwindows
rangingfrom10to40minutes,whileadmissionthresholdsremainunchangedacrossallsettings.
write-budgetconsumption,andSSDreaddemand.
| | | | | | | | Base | +Meta | +Promotion | +Overlap | +Coordinator | |
| --- | --- | --- | --- | --- | --- | --- | ---- | -------- | ---------- | -------------------- | ------------ | --- |
| | | | | | | | | TTFT Avg | | Transfer P25 Bad Avg | | |
PerformanceunderVaryingPressure.Weexaminewhether
| | | | | | | ycnetal dezilamroN | 0.8s -1.3% -2.7% | 1.5s | -0.5% -3.3% | -4.2%0.4 s | -3.3%1.1 | s |
| --- | --- | --- | --- | --- | --- | ------------------ | ---------------- | ---- | ----------- | ---------- | -------- | --- |
afixedevictionthresholdcanadaptSSDpersistencetochang- 1.0 -4.3% -6.4% -3.8% -5.8%
11.7 %
ing serving load. We replay the earliest 100 sessions with 0.9 - %
| | | | | | | | | | | | 7.1% | 16.2 7.7% |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | --------- |
theirstartsdistributedover10-,20-,30-,or40-minutewin- -1 - -1
| | | | | | | 0.8 | | | | | | -26.7% |
| ----- | ------------ | ------ | -------------- | ---------------- | --- | --- | --- | --- | --- | --- | --- | ------ |
| dows. | All settings | retain | the same 1,960 | requests without | | | | | | | | |
-31.9%
| truncationanduse0.5M-tokenDRAM,5M-tokenSSDcapac- | | | | | | 0.7 | | | | | | |
| ------------------------------------------------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
ity.Smallerwindowsincreaseconcurrentdemand.Figure11
0.6
| | | | | | | | Qwen3.5-9B | | Qwen3-8B | Qwen3.5-9B | Qwen3-8B | |
| --------------------------------- | ----------------------------------------- | --- | -------------------- | --- | --- | --------- | ------------------ | --- | -------- | ---------------------- | -------- | --- |
| compareseviction-basedadmission(θ | | | E =2),reuse-gatedad- | | | | | | | | | |
| mission(θ | =4),andfullduplicatecaching,withadmission | | | | | | | | | | | |
| | R | | | | | Figure12: | Cumulativeablation | | | of SARIA’sreloadmecha- | | |
thresholdsfixedacrossallloadlevels.
nismsunderfullduplicatecaching.MeanTTFTisnormalized
Eviction-basedadmissionreducespersistenceasserving
toBase,whilemeantransfertimeovertheslowest25%of
pressurefalls.Expandingthesession-startwindowfrom10to
requestsisnormalizedto+Meta.
40minutesreducesSSDwritesby28.3%forQwen3.5-9Band
23.5%forQwen3-8B,whileimprovingP25badCHAR.Mean
TTFTalsovariesoveranarrowerrangethanwithreuse-gated furtherreducesthistailmetricby17.8%and11.0%,respec-
admissionforbothmodels.Withafixedthreshold,eviction tively.Sourceplacementandtransferschedulingcontribute
historythusadaptspersistencetoload:higherpressuremakes together,withthelargestrelativegainsincache-transferla-
| more | blocks qualify | for SSD | retention, while | lighter load | | tency. | | | | | | |
| ---- | -------------- | ------- | ---------------- | ------------ | --- | ------ | --- | --- | --- | --- | --- | --- |
allowsDRAMtocovermorereuseandconservesSSDwrites.
AblationStudyforSARIADesign.Weisolatethereload 5.3 MicroExperiments
| mechanisms | using | the 350-session | workload | with full | du- | | | | | | | |
| ---------- | ----- | --------------- | -------- | --------- | --- | --- | --- | --- | --- | --- | --- | --- |
plicatecachingandonein-flightrequest.Startingfromper- Disk Admission Trade-offs. Figure 13 evaluates the two
object metadata and serial tier loading (Base),we cumula- admissionsignalsonA30usingTraceLabandWEKA.We
tivelyaddunifiedmetadataandmixed-sourcelookup(+Meta;
varytheevictionthreshold,thereuse-onlythreshold,andtheir
§4.3),partialDRAMpromotion(§4.4),tieroverlap,andco- combinationwhileretainingthesamecachecapacitiesand
ordination(§4.5).Figure12normalizesmeanTTFTtoBase
reloadmechanisms.ThedisplayedevictionthresholdsareE∈
and P25 bad transfertime to +Meta. Since Meta increases {2,3,4,6}; reuse-only thresholds are R∈{4,8,16,32,48};
cachehitsandthetransferredvolume,Basetransfertimeis combinedsettingsfixE =2anduseR∈{2,8,24},withan
shownonlyforreference.
| | | | | | | additional | R=48 | setting | evaluated | only | on WEKA. | Full |
| --- | --- | --- | --- | --- | --- | ---------- | ---- | ------- | --------- | ---- | -------- | ---- |
ThecompleteconfigurationreducesmeanTTFTby6.4% duplicate caching provides a reference with no admission
| forQwen3.5-9Band5.8%forQwen3-8BrelativetoBase,and | | | | | | filtering. | | | | | | |
| ------------------------------------------------- | --- | --- | --- | --- | --- | ---------- | --- | --- | --- | --- | --- | --- |
P25badtransfertimeby31.9%and26.7%relativeto+Meta. Eviction-basedadmissionreachescomparablereusecover-
Addingthecoordinatortotheoverlap-enabledconfiguration agewithsubstantiallyfewerwrites.OnTraceLab,reuse-only
10
Eviction-gated: E∈(2,3,4,6) Combined: E=2∨R∈(2,8,24,48) SARIA w/o Meta SARIA
Full Duplicate Reuse-only: R∈(4,8,16,32,48) TTFT avg CHAR P25 bad avg
| | CHAR P25 bad avg | | | | TTFT (s) P25 bad avg | | | 1.1 | | | | | 78.2%+5.0% | | |
| ---------------- | ---------------- | --- | ---- | --- | -------------------- | --- | --------------- | ---- | --- | ---- | --- | ------------------- | ---------- | ---------- | --- |
| | | | | | | | TFTT dezilamroN | 1.1s | | 4.3s | | RAHC dezilamroN 1.1 | | 82.4%+3.8% | |
| 1.00 | | | R=2 | 5 | | | | 1.0 | | | | | | | |
| baLecarT 5.3newQ | | | | | | | | | | | | 1.0 | | | |
| 0.90 | | E=2 | R=24 | 4 | | | | | | | | | | | |
0.9
| 0.80 | | | | | | | | | | | | 0.9 | | | |
| ---- | --- | --- | --- | --- | --- | --- | --- | --- | ------ | --- | ------ | --- | --- | --- | --- |
| | | | R=4 | 3 | | | | | -25.4% | | -23.5% | | | | |
| 0.70 | E=6 | | | | | | | 0.8 | | | | | | | |
| | | | | 2 | | | | | | | | 0.8 | | | |
0.60
| | R=48 | | | | | | | 0.7 | | | | | | | |
| --- | ---- | --- | ----- | --- | ------- | ----- | --- | ------------ | --- | ---------- | --- | ------------ | --- | ---------- | --- |
| | 0.4× | 1× | 2× 4× | | 0.4× 1× | 2× 4× | | | | | | 0.7 | | | |
| | | | | | | | | Qwen3.5-5.6B | | Qwen3-1.7B | | Qwen3.5-5.6B | | Qwen3-1.7B | |
20
| 0.96 | | | R=2 | | | | | | | | | | | | |
| -------------- | --- | --- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| baLecarT 3newQ | | | R=24 | | | | | | | | | | | | |
0.88 E=2 16 Figure 14: Comparison of unified context metadata and
| 0.80 | | | R=4 | 12 | | | | | | | | | | | |
| ---- | --------- | ---- | ----- | --- | --------- | -------- | ---------------------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- |
| | E=6 | | | | | | mixed-sourcelookupunderE2admission.Allvaluesarenor- | | | | | | | | |
| 0.72 | | | | 8 | | | malizedtotheconfigurationwithbothmechanismsdisabled. | | | | | | | | |
| | 0.2× R=48 | 0.4× | 1× 2× | 4× | 0.2× 0.4× | 1× 2× 4× | | | | | | | | | |
R=2
| 0.96 | | E=2 R=48 | | 6.0 | | | | | | TTFT | | Transfer | Disk read | | |
| --------- | --- | -------- | --- | --- | --- | --- | --- | --- | ---------- | ---- | --- | -------- | --------- | --- | --- |
| 5.3newQ | | | R=4 | | | | | | | | | | | | |
| akeW 0.90 | | | | | | | | | Qwen3.5-9B | | | | Qwen3-8B | | |
4.5
| | | | | | | | eulav dezilamroN | | 0.723s | -3.5% | -3.4% | 1.501s | | | |
| ---- | --- | --- | --- | --- | --- | --- | ---------------- | ---- | ------ | ------ | ------ | ------ | --- | --- | ----- |
| 0.84 | E=6 | | | 3.0 | | | | 1.00 | | | | | | | -9.0% |
| 0.78 | | | | 1.5 | | | | | | | | | | | |
| | | | | | | | | 0.75 | 0.155s | -17.9% | -17.3% | 0.455s | | | |
R=48
| | | 1× | 2× 4× | | 1× | 2× 4× | | 0.50 | 6.158 TiB | | | 22.114 TiB | | | -31.8% |
| --- | --- | --- | ----- | --- | --- | ----- | --- | ---- | --------- | --- | --- | ---------- | --- | --- | ------ |
Cumulative disk writes / disk capacity
0.25
| | | | | | | | | | | | -80.1% | | | | -82.6% |
| --------- | ------------------------------------------ | --- | --- | --- | --- | --- | --- | ---- | --- | ------- | ------ | --- | ------- | --- | ------ |
| Figure13: | Comparisonamongeviction-based,reuse-based, | | | | | | | 0.00 | | | | | | | |
| | | | | | | | | 0 | 0.2 | 0.4 0.6 | 0.8 1 | 0 | 0.2 0.4 | 0.6 | 0.8 1 |
andcombinedadmissionpoliciesonTraceLabandWEKA Promotion rate r
usingtheA30testbed.
| | | | | | | | Figure | 15: | Promotion-ratio | | | sweep | under | full | duplicate |
| --- | --- | --- | --- | --- | --- | --- | ------ | --- | --------------- | --- | --- | ----- | ----- | ---- | --------- |
cachingwithcoordinatedreload.Metricsarenormalizedto
R=4requiresabout1.9timesasmanywritesasE2forsimi- thezero-promotionconfiguration(r=0).
larP25badCHARonbothmodels.Addingthereusebranch
withR=2toE2furtherimprovescoverageforpopularcon-
| | | | | | | | independently | | | to theirr=0 | values. | | Both r=1 | points | use |
| --- | --- | --- | --- | --- | --- | --- | ------------- | --- | --- | ----------- | ------- | --- | -------- | ------ | --- |
textsatgreaterwritecost.Together,thesignalsexposearange
laterimplementationsnapshotsandareincludedasreferences;
| ofwrite–coverage | | trade-offs,withlowerP25 | | | badTTFT | as | | | | | | | | | |
| ---------------- | --- | ----------------------- | --- | --- | ------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
morereusableKVcacheisretained. comparisonsbelowuser≤0.8fromacommonsnapshot.
PromotionreducesSSDtrafficandacceleratessubsequent
UnifiedMetadataandReloadPlanning.Weevaluateuni-
| | | | | | | | reloads. | | For Qwen3.5-9B, | | r = | 0.6 | reduces | SSD | reads by |
| --- | --- | --- | --- | --- | --- | --- | -------- | --- | --------------- | --- | --- | --- | ------- | --- | -------- |
fiedcontextmetadataandmixed-sourcereloadplanningby
| | | | | | | | 43.8%, | mean | transfer | time | by | 17.9%, | and | mean | TTFT by |
| --- | --- | --- | --- | --- | --- | --- | ------ | ---- | -------- | ---- | --- | ------ | --- | ---- | ------- |
enablingordisablingthemtogetherunderE2admissionon 3.5%. For Qwen3-8B,r=0.8 yields reductions of 61.7%,
A30.Thecomparisonholdspromotion,overlap,coordination,
31.6%,and9.0%,respectively.OnQwen3.5-9B,increasing
andcachecapacitiesfixed.
rfrom0.6to0.8furtherreducesSSDreadsbutdoesnotim-
Figure14showsthatenablingthesemechanismsreduces
provelatency.Thisagreeswiththebandwidthbalancein§4.4:
mean TTFT by 25.4% for Qwen3.5-5.6B and 23.5% for DRAMcancomplementSSDbandwidthwithoutholdingthe
Qwen3-1.7B,whileimprovingP25badCHARforbothmod- entirereloadset,soadditionalreadsavingsneednotshorten
els.
reloads.
Thelossofreusewithoutthesemechanismsstemsinpart
CoordinatedMulti-SourceReload.Weisolatetransfercon-
fromLMCache’stier-prefixlookup,whichcannotextenda
| | | | | | | | tention | on | A30 | with a | single | synthetic | request | using | the |
| --- | --- | --- | --- | --- | --- | --- | ------- | --- | --- | ------ | ------ | --------- | ------- | ----- | --- |
prefixwithDRAM-onlyblocksafterswitchingtoSSD.Se-
| | | | | | | | Qwen3-8B | | KV | layout | and no | model | computation. | | The re- |
| --- | --- | --- | --- | --- | --- | --- | -------- | --- | --- | ------ | ------ | ----- | ------------ | --- | ------- |
lectiveadmissioncanproduceDRAM–SSD–DRAMprefixes,
questreloads135,168cachedtokens,splitequallybetween
soomittingthefinalsegmentforcesrecomputationofexist-
DRAMandSSD(9.28125GiBpertier).Figure16compares
ingKVcacheblocks.Unifiedreloadplanningrecoversthese
serialloading,uncoordinatedoverlap,andcoordinatedover-
interleavedhitsandreducesredundantprefill(§4.3).
| | | | | | | | lap | reloadfora | | slowerSSD,a | | fasterSSD,andtwo | | | striped |
| --- | --- | --- | --- | --- | --- | --- | --- | ---------- | --- | ----------- | --- | ---------------- | --- | --- | ------- |
LazyPartialDRAMPromotion.Toisolatethebenefitofpro- SSDs. Thecoordinatorusesa24GB/sshared-ingressbud-
motion,wefixfullduplicatecachingandcoordinatedreload getandrespectiveGDSallowancesof8,12,and23.9GB/s.
and vary the target DRAM share r, using the 350-session SSDtrafficisreconstructedfromDMAevents;DRAMtraffic
workload with one in-flight request. Figure 15 normalizes isestimatedfromPCIereceivetrafficaftersubtractingSSD
meanTTFT,meancache-transfertime,andSSDreadvolume trafficandnormalizedtotheknownDRAMpayload.Each
11
DRAM estimate SSD to GPU Completion andCachedAttention [9] extendKV caching beyondGPU
memoryusinghostDRAMandSSDs.HiCachereloadsSSD-
| | | GDS 8 GB/s | GDS 12 GB/s | GDS 24 GB/s | | | | | | | | |
| --- | --- | ---------- | ----------- | ----------- | --- | --- | --- | --- | --- | --- | --- | --- |
24
| | | | | | resident | KV cache | through | DRAM | | and supports | | full pro- |
| --- | --- | --- | --- | --- | -------- | -------- | ------- | ---- | --- | ------------ | --- | --------- |
laireS
| | 16 | | | | motion,whereasFlexKV[8]andDynamoKVBM[19]use | | | | | | | |
| --- | --- | --- | --- | --- | ---------------------------------------------- | --- | --- | --- | --- | --- | --- | --- |
| | 8 | | | | GDStoreloadSSDhitsdirectlyintoGPUmemorywithout | | | | | | | |
DRAMpromotion.Tutti[23]exploitsSSDparallelismand
| | 0 | 0.6 1.2 1.8 | 0 0.4 0.8 | 1.2 0 0.3 0.6 0.9 | | | | | | | | |
| --- | --- | ----------- | --------- | ----------------- | ------------ | --- | ----- | ----------- | --- | ----------- | --- | ----- |
| | | | | | an optimized | I/O | stack | to approach | | DRAM-backed | | cache |
)s/BG( ciffarT 24
| | palrevO | | | | performance.UnifiedKVPooling[13]partitionsKVcache | | | | | | | |
| --- | ------- | --- | --- | --- | ------------------------------------------------- | --- | --- | --- | --- | --- | --- | --- |
16
accordingtodevicebandwidth,whileHiFC[12]combines
| | 8 | | | | GDSwithpseudo-SLCregions.Strata[36]andSwarm[33] | | | | | | | |
| --- | --- | ----------- | --------- | ----------------- | ----------------------------------------------- | ------- | -------------- | --- | --- | --------- | ---------- | --- |
| | | | | | improve | storage | I/O scheduling | | and | multi-SSD | placement. | |
| | 0 | 0.6 1.2 1.8 | 0 0.4 0.8 | 1.2 0 0.3 0.6 0.9 | | | | | | | | |
Incontrast,SARIAtargetsdeploymentswithlimitedSSDen-
24
rotanidrooC
duranceandparallelism,usingpartialDRAMpromotionand
16
coordinatedmixed-sourcereloadstoreducedependenceon
| | 8 | | | | thebandwidth-limitedSSDpathwithoutexcessiveDRAM | | | | | | | |
| --- | --- | --- | --- | --- | ----------------------------------------------- | --- | --- | --- | --- | --- | --- | --- |
competition.
| | 0 | 0.6 1.2 1.8 | 0 0.4 0.8 | 1.2 0 0.3 0.6 0.9 | | | | | | | | |
| --- | --- | ----------- | --------- | ----------------- | --- | --- | --- | --- | --- | --- | --- | --- |
Time from payload transfer start (s) SSD Admission and Write Endurance. Flashield [6] re-
ducesflashwritesbyadmittinggeneral-purposekey–value
Figure 16: Comparison ofcoordinatedand uncoordinated objects based on their observed reads and updates, while
reloadtrafficontheA30testbedwithequalDRAMandSSD CacheSack [37] and Baleen [34] similarly optimize flash-
payloadsunderdifferenthardware. cacheadmission.Thesepoliciestargetgeneralstoragework-
| | | | | | loads rather | than | the session-level | | reuse | structure | | of agent |
| --- | --- | --- | --- | --- | ------------ | ---- | ----------------- | --- | ----- | --------- | --- | -------- |
serving.SARIAexploitsthisstructurebydistinguishingreuse
plotcontainsonemeasuredreloadin20-msbins,timedfrom
alreadycoveredwithinaDRAMresidencyfromreusethatex-
payloadsubmission. tendsacrossDRAMresidencies.Repeatedevictionandreuse
Forbothsingle-SSDconfigurations,uncoordinatedoverlap thereforeprovideanadmissionsignalthatreflectsbothses-
leavesSSDprogresslowduringtheinitialDRAMburstand
sionreuseandcachecompetition,complementedbyareuse
providesnoimprovementoverserialloading.Coordination thresholdforhighlypopularprefixes.
allowsbothsourcestoprogress,reducingcompletiontimeby
18.0%ontheslowerSSDand20.7%onthefasterSSDrela-
7 Conclusions
tivetoserialloading.WithtwoSSDs,however,coordination
increasescompletiontimeby15.8%relativetoserialloading.
SSD-backedKVcachingfacestwopracticalchallenges:ex-
ThediskallowanceleaveslittlebandwidthforDRAM,while
cessivewritesrapidlyconsumeSSDendurance,whilelimited
coordinatoroverheadpreventsSSDtrafficfromreachingthat
| | | | | | SSD parallelism | | constrains | reload | bandwidth. | | We | present |
| --- | --- | --- | --- | --- | --------------- | --- | ---------- | ------ | ---------- | --- | --- | ------- |
allowance.CoordinationthereforehelpsunderlimitedSSD
SARIA,ahierarchicalKVcachingsystemthataddressesthese
bandwidth,butitsallowancesmustmatcheffectivepathband-
widths(§4.5). challengesthroughPersistence-AwareSSDAdmission,which
| | | | | | uses reuse | and eviction | | history | to avoid | unnecessary | | SSD |
| --- | --- | --- | --- | --- | ---------- | ------------ | --- | ------- | -------- | ----------- | --- | --- |
writes;LazyPartialDRAMPromotion,whichretainsselected
6 RelatedWork
| | | | | | SSD-hit | KV cache | blocks | in DRAM | | for subsequent | | reuse |
| --- | --- | --- | --- | --- | -------------- | -------- | ------------ | ------- | -------------------- | -------------- | --- | ----- |
| | | | | | while limiting | DRAM | competition; | | andcoordinatedmulti- | | | |
KV cache management. SGLang [42] organizes reusable sourcereload,whichschedulesDRAMandGDStransfersto
prefixes with radix trees, while HiCache [28] further uses efficientlyshareGPUingressbandwidth.AcrossTraceLab-
derivedworkloadsandbothfull-attentionandhybrid-attention
| access | counts | to filter | HBM-to-DRAM | backups. Unified | | | | | | | | |
| ------ | ------ | --------- | ----------- | ---------------- | --- | --- | --- | --- | --- | --- | --- | --- |
Radix Cache [11] extends prefix management to hybrid- models,SARIAreducesSSDwritesby57.1–58.9%compared
attention models,andMarconi [20] introduces checkpoint- with full duplicate caching while reducing mean TTFT by
42.3–79.7%comparedwithDRAM-onlycaching.
awarecachingforrecurrentstates.SAECache[7]usesevic-
tionfeedbacktoestimatetokenreusevalue.Thesetechniques
| improvecacheorganizationandadmissionwithinmemoryhi- | | | | | References | | | | | | | |
| --------------------------------------------------- | --- | --- | --- | --- | ---------- | --- | --- | --- | --- | --- | --- | --- |
erarchies,butdonotexplicitlyaddressSSDendurance.SARIA
insteadusesreuseandevictionhistoryacrossDRAMresiden-
| | | | | | [1] Amazon | Web | Services. | | New—amazon | | EC2 | P5 in- |
| --- | --- | --- | --- | --- | ---------- | --- | --------- | --- | ---------- | --- | --- | ------ |
ciestodeterminewhichKVcacheblockswarrantpersistence
stancespoweredbyNVIDIAH100TensorCoreGPUs
onSSD.
| | | | | | for | accelerating | generative | | AI | and HPC | applications. | |
| --- | --- | --- | --- | --- | --- | ------------ | ---------- | --- | --- | ------- | ------------- | --- |
SSD-backedKVcaching.Mooncake[17,22],LMCache[3], AWSNewsBlog,2023.
12
[2] Shai Bergman,Tanya Brokhman,Tzachi Cohen,and [13] MinchulKang,ChangyongShin,JinwooJeong,Jaerim
MarkSilberstein. SPIN:Seamlessoperatingsystemin- Park, Woohyun Kim, Bonyul Gu, Dongwoo Kang,
tegrationofpeer-to-peerDMAbetweenSSDsandGPUs. Gyeongsik Yang,and Chuck Yoo. Unified KV pool-
ACMTransactionsonComputerSystems,36(2):1–26, ingtoacceleratelong-contextLLMserving,2026.
2019.
| | | | | | | | [14] KIOXIA | | Corporation. | | CD8P-R | series | data center |
| --------- | ------ | ---------- | ----- | ---------- | --- | ------- | ----------- | -------------- | ------------ | ---- | ------- | ------ | ----------- |
| | | | | | | | NVMe | read-intensive | | SSD. | Product | brief, | Rev. 1.1, |
| [3] Yihua | Cheng, | Yuhan Liu, | Jiayi | Yao, Yuwei | | An, Xi- | | | | | | | |
aokun Chen,Shaoting Feng,Yuyang Huang,Samuel 2024. TLC;1DWPD;five-yearwarranty.
| Shen,Kuntai | | Du,and | Junchen | Jiang. | LMCache: | An | | | | | | | |
| ----------- | --- | ------ | ------- | ------ | -------- | --- | ------------ | ---------------------------------- | --- | --- | --- | --- | --- |
| | | | | | | | [15] Lenovo. | ThinkSystemSR680aV3withB200product | | | | | |
efficientKVcachelayerforenterprise-scaleLLMinfer-
| | | | | | | | guide. | LenovoPress,2026. | | | | | |
| --- | --- | --- | --- | --- | --- | --- | ------ | ----------------- | --- | --- | --- | --- | --- |
ence,2025.
| | | | | | | | [16] Micron | Technology,Inc. | | | Comparing | SSD | and HDD |
| ------------------------ | ---------------------------- | ------------------------------ | ---------------------- | --- | --- | --- | --------------------------- | --------------- | --- | ------------------------------ | --------- | ----------------- | ------- |
| [4] FloydChristofferson. | | Activatinguntappedtier0storage | | | | | | | | | | | |
| | | | | | | | enduranceintheageofQLCSSDs. | | | | | Whitepaper. | |
| withinyourGPUservers. | | | HammerspaceWhitePaper, | | | | | | | | | | |
| 2026. | Version1.2,February2026,p.5. | | | | | | | | | | | | |
| | | | | | | | [17] MooncakeProject. | | | MooncakeStoreSSDoffloaddesign. | | | |
| | | | | | | | Mooncakedocumentation,2026. | | | | | AccessedSeptember | |
| [5] DaoCloudTeam. | | Fromday0toproductionSLAs:Serv- | | | | | | | | | | | |
3,2026.
ingGLM-5.2on24NVIDIAB300GPUswithvLLM.
| vLLMBlog,July2026. | | | PublishedJuly23,2026;8K- | | | | | | | | | | |
| ------------------ | --- | --- | ------------------------ | --- | --- | --- | ----------- | --- | ------------ | --- | --------- | ------- | ------ |
| | | | | | | | [18] NVIDIA | | Corporation. | | GPUDirect | Storage | Design |
inputTP1DP2EPprefillpoint;accessedSeptember9,
| | | | | | | | Guide. | Online. | AccessedSeptember2026. | | | | |
| --- | --- | --- | --- | --- | --- | --- | ------ | ------- | ---------------------- | --- | --- | --- | --- |
2026.
| | | | | | | | [19] NVIDIA | | Dynamo | Project. | Dynamo | KVBM: | Host |
| --- | --- | --- | --- | --- | --- | --- | ----------- | --- | ------ | -------- | ------ | ----- | ---- |
[6] Assaf Eisenman, Asaf Cidon, Evgenya Pergament, and disk cache reload implementation. Source code.
OrHaimovich,RyanStutsman,MohammadAlizadeh,
Sourcesnapshotatcommit4b72f79c,inspectedSeptem-
| andSachinKatti. | | Flashield:Ahybridkey-valuecache | | | | | ber2026. | | | | | | |
| ------------------------------------ | --- | ------------------------------- | --- | ------------ | --- | --- | -------- | --- | --- | --- | --- | --- | --- |
| thatcontrolsflashwriteamplification. | | | | In16thUSENIX | | | | | | | | | |
SymposiumonNetworkedSystemsDesignandImple- [20] RuiPan,ZhuangWang,ZhenJia,CanKarakus,Luca
mentation(NSDI19),pages65–78,2019. Zancato,TriDao,YidaWang,andRaviNetravali. Mar-
| | | | | | | | coni: | Prefix | caching | for | the era of | hybrid | LLMs. In |
| --- | --- | --- | --- | --- | --- | --- | ----- | ------ | ------- | --- | ---------- | ------ | -------- |
[7] Shaoke Fang,Ziang Li,Wenfei Wu,Jiatong Ji,Qing- ProceedingsofMachineLearningandSystems(MLSys),
| song | Liu, | and Ruizhi | Pu. Not | all tokens | are | worth | | | | | | | |
| ---- | ---- | ---------- | ------- | ---------- | --- | ----- | --- | --- | --- | --- | --- | --- | --- |
volume7,2025.
| caching: | Learning | semantic-aware | | eviction | for | LLM | | | | | | | |
| -------- | -------- | -------------- | --- | -------- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
prefixcaches,2026. [21] Ruoyu Qin, Weiran He, Yaoyu Wang, Zheming Li,
XinranXu,YongweiWu,WeiminZheng,andMingx-
| [8] FlexKV | Project. | FlexKV: | GDS | cache | reload | imple- | | | | | | | |
| ---------- | -------- | ------- | --- | ----- | ------ | ------ | --- | ------ | --------------------- | --- | --- | ------- | -------- |
| | | | | | | | ing | Zhang. | Prefill-as-a-service: | | | KVCache | of next- |
mentation. Source code. Source snapshotatcommit generationmodelscouldgocross-datacenter,2026. Ver-
40b8c7fc,inspectedSeptember2026.
sion1,Table3;accessedSeptember11,2026.
[9] BinGao,ZhuominHe,PuruSharma,QingxuanKang,
| | | | | | | | [22] Ruoyu | Qin,ZhemingLi,Weiran | | | | He,Jialei | Cui,Feng |
| --- | --- | --- | --- | --- | --- | --- | ---------- | -------------------- | --- | --- | --- | --------- | -------- |
DjordjeJevdjic,JunboDeng,XingkunYang,ZhouYu, Ren,Mingxing Zhang,Yongwei Wu,Weimin Zheng,
andPengfeiZuo. Cost-efficientlargelanguagemodel andXinranXu. Mooncake:Tradingmorestoragefor
servingformulti-turnconversationswithCachedAtten-
lesscomputation—aKVCache-centricarchitecturefor
tion. In 2024 USENIX Annual Technical Conference servingLLMchatbot. In23rdUSENIXConferenceon
(USENIXATC24),pages111–126,2024. FileandStorageTechnologies(FAST25),pages155–
170,2025.
| [10] GIGABYTE. | | G593-SD1-AAX3GPUserver. | | | | Product | | | | | | | |
| -------------- | --- | ----------------------- | --- | --- | --- | ------- | --- | --- | --- | --- | --- | --- | --- |
specifications. AccessedSeptember2026. [23] ShiQiu,YifanHu,XintaoWang,WenhaoZhu,Jianqin
| | | | | | | | Yan,Hao | | Chen,Kaiqiang | | Xu,Kai | Chen,and | Yiming |
| --- | --- | --- | --- | --- | --- | --- | ------- | --- | ------------- | --- | ------ | -------- | ------ |
[11] ZhanghengHuang,KeBao,YiZhang,JialinOuyang,
| | | | | | | | Zhang. | Tutti:MakingSSD-backedKVcachepractical | | | | | |
| --- | ------- | ------------ | ----- | ------ | --- | -------- | ------ | -------------------------------------- | --- | --- | --- | --- | --- |
| and | Sicheng | Pan. Unified | radix | cache: | One | tree for | | | | | | | |
forlong-contextLLMserving,2026.
| hybridmodelprefixcaching. | | | LMSYSOrgBlog,2026. | | | | | | | | | | |
| ------------------------- | --- | --- | ------------------ | --- | --- | --- | -------------- | --- | ---------------------------------- | --- | --- | --- | --- |
| | | | | | | | [24] QwenTeam. | | Qwen3-235B-A22B-Instruct-2507-FP8. | | | | |
[12] InhoJeong,SunghyeonWoo,SolNamkung,andDong-
HuggingFacemodelcardandconfiguration,2025.
| sukJeon. | HiFC:High-efficiencyflash-basedKVcache | | | | | | | | | | | | |
| -------- | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
swappingforscalingLLMinference. InAdvancesin [25] Qwen Team. Qwen3-30B-A3B-Instruct-2507-FP8
Neural Information Processing Systems 38 (NeurIPS modelconfiguration. HuggingFacemodelrepository,
| 2025),2025. | | | | | | | 2025. | AccessedSeptember11,2026. | | | | | |
| ----------- | --- | --- | --- | --- | --- | --- | ----- | ------------------------- | --- | --- | --- | --- | --- |
13
[26] SemiAnalysis. WekaTrace:Claudecodeproxytraces. [37] Tzu-WeiYang,SethPollen,MustafaUysal,ArifMer-
HuggingFacedataset,2026. AccessedSeptember2026. chant,andHomerWolfmeister. CacheSack:Admission
| | | | | | | | optimizationforgoogledatacenterflashcaches. | | | In2022 |
| --- | --- | --- | --- | --- | --- | --- | ------------------------------------------- | --- | --- | ------ |
[27] SGLang Project. Add HPC-Ops attention backend. USENIXAnnualTechnicalConference(USENIXATC
GitHub pull request and benchmark artifact, 2026. 22),pages1021–1036,2022.
| Mergedpull | | request | #30540; | accessedSeptember11, | | | | | | |
| ---------- | --- | ------- | ------- | -------------------- | --- | --- | --- | --- | --- | --- |
[38] Gyeong-InYu,JooSeongJeong,Geon-WooKim,Soo-
2026.
| | | | | | | | jeongKim,andByung-GonChun. | | Orca:Adistributed | |
| --- | --- | --- | --- | --- | --- | --- | -------------------------- | --- | ----------------- | --- |
[28] SGLangProject. HiCachesystemdesignandoptimiza- servingsystemforTransformer-Basedgenerativemod-
tion. SGLangdocumentation,2026. AccessedSeptem- els. In16thUSENIXSymposiumonOperatingSystems
| ber1,2026. | | | | | | | DesignandImplementation(OSDI22),pages521–538, | | | |
| ---------- | --- | --- | --- | --- | --- | --- | --------------------------------------------- | --- | --- | --- |
2022.
| [29] SGLangProject. | | Q8KV8FP8SparsePrefillonGLM- | | | | | | | | |
| ------------------- | ------------------ | --------------------------- | --- | ------ | ------------- | ----- | -------------------------------------- | --- | --- | ------------- |
| | | | | | | | [39] LingfanYu,JinkunLin,andJinyangLi. | | | Statefullarge |
| 5.2 | and DeepSeek-V3.2. | | | GitHub | pull request, | 2026. | | | | |
Merged pull request #31888; accessed September 8, languagemodelservingwithPensieve. InProceedings
| | | | | | | | of the Twentieth | European Conference | | on Computer |
| --- | --- | --- | --- | --- | --- | --- | ---------------- | ------------------- | --- | ----------- |
2026.
Systems(EuroSys),2025.
[30] ZhaoyuanSu,OlatunjiRuwase,KarthikGanesan,Au-
| | | | | | | | [40] Z.ai. GLM-5.2-FP8. | Hugging | Face | model card and |
| --- | --- | --- | --- | --- | --- | --- | ----------------------- | ------- | ---- | -------------- |
rickQiao,SamyamRajbhandari,JunchengYang,Yue
configuration,2026.
| Cheng,andYuxiongHe. | | | MoE-Prefill:Zeroredundancy | | | | | | | |
| ------------------- | --- | --- | -------------------------- | --- | --- | --- | --- | --- | --- | --- |
overheadsinMoEprefillserving,2026.
[41] ChenZhang,KuntaiDu,ShuLiu,WoosukKwon,Xi-
angxiMo,YufengWang,XiaoxuanLiu,KaichaoYou,
[31] SyFILab.TraceLab:Codingagenttracedataset.Project
ZhuohanLi,MingshengLong,JidongZhai,JosephGon-
| website,2026. | | AccessedSeptember2026. | | | | | | | | |
| ------------- | --- | ---------------------- | --- | --- | --- | --- | ----------------------------------- | ---------------------------- | --- | ---------- |
| | | | | | | | zalez,andIonStoica. | Jenga:Effectivememorymanage- | | |
| | | | | | | | mentforservingLLMwithheterogeneity. | | | InProceed- |
[32] ShubhamTiwari,TapanChugh,NashRickert,Simon
ingsoftheACMSIGOPS31stSymposiumonOperating
| Peter,RatulMahajan,andHaiyingShen. | | | | | | CacheWise: | | | | |
| ---------------------------------- | --- | --- | --- | --- | --- | ---------- | --- | --- | --- | --- |
SystemsPrinciples(SOSP),pages446–461,2025.
| Understanding | | workloads | | and | optimizing | KVCache | | | | |
| ------------- | --- | --------- | --- | --- | ---------- | ------- | --- | --- | --- | --- |
managementforefficientlyservingLLMcodingagents,
[42] LianminZheng,LiangshengYin,ZhiqiangXie,Chuyue
2026.
| | | | | | | | Sun, Jeff Huang, | Cody Hao | Yu, Shiyi | Cao, Christos |
| --- | --- | --- | --- | --- | --- | --- | ---------------- | -------- | --------- | ------------- |
Kozyrakis,IonStoica,JosephE.Gonzalez,ClarkBar-
| [33] Tuowei | Wang, | Liyun | Chu, | Ruwen | Fan, | and Ju Ren. | | | | |
| ----------- | ----- | ----- | ---- | ----- | ---- | ----------- | ------------------ | --------------------------- | --- | --- |
| | | | | | | | rett,andYingSheng. | SGLang:Efficientexecutionof | | |
Swarm:Co-activationawareKVCacheoffloadingacross
structuredlanguagemodelprograms,2023.
multipleSSDs,2026.
[43] KanZhu,MathewJacob,ChenxiMa,YiPan,Stephanie
[34] DanielLin-KitWong,HaoWu,CarsonMolder,Sathya
| | | | | | | | Wang,ArvindKrishnamurthy,andBarisKasikci. | | | Trace- |
| ---------- | --- | ----- | --- | ------ | --------- | ------- | ----------------------------------------- | --- | --- | ------ |
| Gunasekar, | | Jimmy | Lu, | Snehal | Khandkar, | Abhinav | | | | |
Lab:CharacterizingcodingagentworkloadsforLLM
Sharma,DanielS.Berger,NathanBeckmann,andGre-
serving,2026.
| goryR.Ganger. | | Baleen:MLadmission&prefetching | | | | | | | | |
| --------------- | --- | ------------------------------ | --- | --- | --- | --- | --- | --- | --- | --- |
| forflashcaches. | | In22ndUSENIXConferenceonFile | | | | | | | | |
andStorageTechnologies(FAST24),pages347–371,
2024.
[35] YuxingXiang,XueLi,KunQian,YanZhang,Wenyuan
| Yu,EnnanZhai,XinJin,andJingrenZhou. | | | | | | ServeGen: | | | | |
| ----------------------------------- | --- | --- | --- | --- | --- | --------- | --- | --- | --- | --- |
Workloadcharacterizationandgenerationoflargelan-
| guagemodelservinginproduction. | | | | | In23rdUSENIX | | | | | |
| ------------------------------ | --- | --- | --- | --- | ------------ | --- | --- | --- | --- | --- |
SymposiumonNetworkedSystemsDesignandImple-
mentation(NSDI26),pages1845–1859,Renton,WA,
May2026.USENIXAssociation.
| [36] Zhiqiang | | Xie, Ziyi | Xu, | Mark | Zhao, | Yuwei An, | | | | |
| ------------- | --- | --------- | --- | ---- | ----- | --------- | --- | --- | --- | --- |
VikramSharmaMailthody,ScottMahlke,MichaelGar-
| land,andChristosKozyrakis. | | | | Strata:Hierarchicalcon- | | | | | | |
| -------------------------- | --- | --- | --- | ----------------------- | --- | --- | --- | --- | --- | --- |
textcachingforlongcontextlanguagemodelserving,
2025.
14