← Back to summary
Rethinking Query Optimization for Multi-Agent Systems
Converted from PDF with markitdown. Layout may be rough.
Rethinking Query Optimization for Multi-Agent Systems
ZoiKaoudi IoanaGiurgiu
ITUniversityofCopenhagen IBMResearchEurope
Denmark Switzerland
zoka@itu.dk igi@zurich.ibm.com
ABSTRACT
resultinginpipelinesthatareofteninefficient,brittle,andunable
Theproliferationoflargelanguagemodels(LLMs)hasaccelerated toadapttoevolvingworkloadrequirements.Inthesesettings,the
theadoptionofagent-baseddatapipelines.Yetcurrentapproaches primaryunitofcomputationisnolongertheoperatorbuttheagent,
remainadhoc,relyingonfixedstructures,predefinedLLMs,andsin- andtheprimaryartifactisnotafixedqueryplanbutadynamically
gleexecutionengines,withoutsystematicoptimizationacrosshet- structuredpipelineofinteractingagents.Thisshiftcallsforanew
erogeneousdatasourcesandengines.ThispaperpresentsNOMA, direction:datasystemsforagents.Wemustdesignsystemswhere
aqueryoptimizationframeworkformulti-agentdatapipelines.We multi-agentpipelinesthemselvesaretheobjectofoptimization.As
arguethatoptimizingagenticpipelinesisafundamentallydifferent aresult,decisionsaboutpipelinetopology,modelselection,and
queryoptimizationproblem,withcentralchallenges:(i)amulti- executionenginescannolongerbemadeindependentlyorahead
dimensionalsearchandobjectivespace,wheretopology,model, oftimebutmustbejointlyoptimizedundercompetingobjectives
andenginechoicesmustbeoptimizedjointlyacrosslatency,cost, suchaslatency,cost,andaccuracy.
andaccuracy;(ii)avariablepipelinetopology;(iii)theco-existence Challenges. Thisperspectiveisanalogoustoearlier“database
ofdiversedatamodels,leavingnocommonoperatoralgebra;and formachinelearning”efforts,whichapplieddatabaseprinciples
(iv)thesignificantcostofexecutingthesepipelines.Ourcontrolled tooptimizeMLpipelines[18,25].However,multi-agentpipelines
experimentoverareal-world10-agentpipelinerevealsextreme introducenewchallenges.First,theoptimizationspaceisinher-
variance(153×cost,5×latency,25%quality)andthatoptimalplans entlymulti-dimensional,requiringjointreasoningovertopology,
areheterogeneousconfigurationsnouserwouldconstructmanu- models,andenginesundercompetingobjectives.Second,pipeline
ally.Ouranalysisofrealdeploymentsconfirmstheseinefficiencies topologyisitselfadecisionvariable,meaningthattheoptimizer
aresystematic.WepresentNOMAasanintegratedoptimization mustdeterminenotonlyhowtoexecuteapipelinebutalsohow
loopinwhichplangeneration,costestimation,runtimerefinement, tostructureit.Third,thecoexistenceofstructureddata,unstruc-
andsemanticcachingreinforceoneanotheracrossexecutions,set- turedtext,embeddings,andstreamingdataeliminatesanycommon
tingacommunity-wideresearchagendaonqueryoptimizationfor operatoralgebra,preventingclassicalpruningtechniques.Finally,
multi-agentsystems. executionisbothexpensiveandstochastic,asLLMcallsincursig-
nificantmonetarycostandvariabilityinoutputquality,making
PVLDBReferenceFormat: naiveexplorationofthesearchspaceinfeasible.
ZoiKaoudiandIoanaGiurgiu.RethinkingQueryOptimizationfor NOMA. Inthispaper,wepresentNOMA,aqueryoptimization
Multi-AgentSystems.PVLDB,20(1):46-52,2026.
frameworkdesignedformulti-agentdatapipelines.NOMAtreats
doi:10.14778/3845598.3845602
agentsasfirst-classunitsofcomputationandextendsclassicalopti-
1 INTRODUCTION mizationprinciples,suchascostmodeling,plangeneration,andre-
optimization,toasettingwhereplansareheterogeneous,stochastic,
Therapidadvancementoflargelanguagemodels(LLMs),together andstructurallydynamic.TheoptimizerproducesPareto-optimal
withtheemergenceoftool-callinginterfacesandagentorchestra- pipelinesthatbalancelatency,cost,andqualityacrossheteroge-
tionframeworks,hasledtotheriseofmulti-agentdatapipelines: neousexecutionenvironments.Ourcontributionsinclude:
compositions of agents that reason, invoke tools, interact with (1) Wequantifythecostofsuboptimalchoicesonareal-world
heterogeneousdatasources,andcollaboratetoexecutecomplex pipeline,showing153×costvariance,5×latencyvariance,anda
data-driventasks.Thesepipelinesareincreasinglyusedinreal- 25%qualitygapacrosstheplanspaceunderafixedtopologyand
worldapplications,rangingfromcustomersupportanalyticsto showthatvaryingtopologyreachesplansbeyondit(Section2).
financialmonitoringandsupply-chainintelligence. (2) Weanalyzemulti-agentpipelinesstemmingfromrealdeploy-
Datasystemsforagents. Despitethegrowingadoptionofsuch ments,showingthatcurrentpracticesleavesubstantialoptimiza-
pipelines,currentapproachestoconstructingmulti-agentpipelines tionpotentialunrealized(Section3).
remainfundamentallyadhoc.Developersrelyonfixedpipeline (3) Weformalizethenovelproblemofoptimizingmulti-agentdata
structures,pre-selectedmodels,andsingleexecutionenvironments, pipelines(Section4).
(4) WepresentNOMA,avisionforaqueryoptimizationframe-
ThisworkislicensedundertheCreativeCommonsBY-NC-ND4.0International workformulti-agentpipelines,organizedaroundfourresearch
License.Visithttps://creativecommons.org/licenses/by-nc-nd/4.0/toviewacopyof challenges:variablemulti-dimensionalsearch,heterogeneouscost
thislicense.Foranyusebeyondthosecoveredbythislicense,obtainpermissionby
emailinginfo@vldb.org.Copyrightisheldbytheowner/author(s).Publicationrights estimation,continuousoptimizationunderstochasticexecution,
licensedtotheVLDBEndowment. andsemanticcachingofpipelineartifacts(Section5).
ProceedingsoftheVLDBEndowment,Vol.20,No.1ISSN2150-8097. Relatedwork.Existingresearchhaslargelyfocusedonthreelines
doi:10.14778/3845598.3845602
of work. The first incorporates LLMs as user-defined functions
46
Table1:PositioningofNOMAagainstpriorwork.
modelsshouldbeused(modelselection),andwhichenginesshould
executeeachtask(engineselection).Wetakeafirststeptowards
Model Agent quantifyingthispotentialwithacontrolledexperiment.
LLM-in-DB
Dimension routing orchestration NOMA
[11,16,20,24,27]
[6,23] [5,7,30,32]
2.2 ExperimentalSetup
Topologyavariable × ✓ × ✓
Modelassignment ✓ × per-op ✓
Queryengineselection × × singleengine ✓ Data. Thepipelineprocessesapproximately2.8GBacrossthefour
Objective cost–quality single partial Pareto deterministicagentsandbetween8Kand80Kinputtokensper
Scope percall pipeline operator pipeline
Runtimeadaptation × limited × continuous stochasticagent,dependingonthetask.
Cross-pipelinereuse × partial* × semantic Engines. Eachdeterministicagentcanbeassignedtooneoffour
*Helium[32]cachesdeterministicoperatoroutputsacrossbatches. engineconfigurations:SparkorPresto,eachwith1or4worker
nodes.Inourexperimentalsetup,eachagenthasahomeengine
withintraditionalqueryprocessingpipelines[11,16,20,24,27], whereitsdataresides(A2,A3,andA8onSpark;A5onPresto).
treatingeachLLMcallasanoperatorwithinanotherwisefixeddata Runningonadifferentenginefamilyincursadatatransferpenalty
system.ThesecondtargetsmodelroutingacrossLLMs,learning proportionaltodatavolume.
todispatcheachcalltothemostcost-effectivemodelunderqual- LLMs. Each stochastic agent can be assigned to one of three
ityconstraints[6,23].Thethirdfocusesonservingefficiencyand
LLMsspanningthecost-qualityspectrum:Llama3.18B(cheap,
orchestrationontopofagentframeworks,optimizingscheduling,
fast,accuracy0.74–0.82),Qwen2.5-72B(moderatecostandaccu-
resourceuse,andexecutionorderacrossLLMcalls[5,7,30,32].In
racy0.89–0.93),andClaudeSonnet3.5(mostaccurateat0.93–0.96
allthree,optimizationtargetsindividualoperatorsorcalls,making
but orders of magnitude more expensive). To account for error
themcheaper,faster,ormoreaccurate,whiletheoverallpipeline
propagation,wemeasurepipeline-levelqualityastheweighted
structureremainsfixedandgroundedinapredefineddatamodel
geometricmeanofper-agentaccuracy,whichweobtainbytask-
andexecutionengine.Arecentperspective[35]documentsredun-
specificevaluationagainstreferencelabels:classificationaccuracy
dancy and inefficiency in agentic pipelines but leaves the joint
forsentimentandurgencydetection,F1forentityextraction,and
optimizationacrossoursuggesteddimensionsunaddressed.Table1
reference-basedscoringforreportgeneration.Theweightsreflect
summarizesthesedistinctions.
eachagent’scontributiontothefinaloutputandaresetuniformly.
Ourvisionisdeliberatelyambitious.Werequireaparadigmshift:
Planspaceintwodimensions. Tomaketheexperimentfeasible,
fromoptimizingqueriesoverfixedplanstocontinuouslysynthe-
wefirstassumeafixedtopology.Aplanisacompleteassignmentof
sizingandadaptingpipelines.Optimizationbecomesapersistent
anenginetoeverydeterministicagentandanLLMtoeverystochas-
systemfunctionratherthanapre-executionphase.Weaimtore-
ticagent,yielding44×36 =186,624plans.Weuniformlysample
definequeryoptimizationformulti-agentsystems,establishthe
10,000andevaluateeachonmonetarycost,end-to-endlatency,and
foundationsforanewclassofqueryoptimizersthatmovebeyond
outputquality.Toaccountforvariance,weevaluateeachsampled
static pipelines toward adaptive and cost-aware execution, and
planthreetimesandreporttheaverage.Sincetopologyisheldfixed,
identifykeyresearchquestionsthatthecommunitymustaddress.
thisspacecoversonlytwoofthethreeoptimizationdimensions;
2 THECOSTOFSUBOPTIMALCHOICES thevariancereportedbelowisthereforealowerboundonwhat
jointoptimizationovertopology,models,andengineswouldreveal.
Webeginbyestablishingthattheoptimizationproblemisboth
Weshowthetopology’sadditionalbenefitattheendofthissection.
severeandunavoidable.Usingareal-worldmulti-agentpipeline,we
showthatthespaceofpossibleexecutionsexhibitsextremevariance
2.3 Results
acrosscost,latency,andquality,andthatoptimalconfigurations
cannotbeconstructedmanually. Figure2showsthethree-dimensionalspaceofcost,latency,and
accuracy.Wemakethreemainobservations.
2.1 Real-worldMulti-agentPipeline
Theplanspaceexhibitsextremevariance. Acrossthe10,000
Considerthefollowingreal-worldrequestfromourcustomers: sampled plans (which explore only model and engine selection
Real-timeCustomerSupportReporting:Findallurgentcasesreported underafixedtopology),monetarycostrangesfrom$0.01to$2.04
bycustomersfromlivechatsinthelast24hours.Generateadetailed (a153×spread),end-to-endlatencyfrom42sto216s(5.1×),and
reportandflagcasesthatneedpriorityhandling. quality from 0.76 to 0.95 (25% gap). Average plans are far from
Thispipelinedecomposesintotenagents(leftpartofFigure1), optimalonanydimension,sodefaultchoicesleadtopooroutcomes.
combiningstructureddata(customerprofilesfromadata lake), TheParetofrontierisrichandnon-trivial. Wefind266Pareto-
unstructureddata(sentimentanalysisandurgencydetection),and optimalplans,representing2.7%ofthesampledspace.Thefrontier
streamingdata(real-timechatmessages).Wedistinguishbetween spansfromacheapestplanat$0.01(42.5slatency,0.77quality)
twoclassesofagents:stochastic,whoseoutputsmayvaryacross to a highest-quality plan at $1.84 (125.9s, 0.95 quality)—a 138×
executions(lightbluenodes),anddeterministic,whichreturnthe costincreasefora19%qualityimprovement,whiletriplingthe
sameoutputforthesameinput(darkbluenodes). latency.Theseconfigurationsarehighlyunintuitive:theyrequire
Thepotentialforoptimizationissubstantialalongthreedimen- fine-grainedcoordinationbetweenmodelselectionandexecution
sions: how the pipeline should be structured (topology), which placementthatdevelopersdonotperforminpractice.
47
| Abstract Agent - Stochastic | Abstract Agent - Deterministic | Executable Agent - LLM call | Executable Agent - Tool call | | | | | | | | |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- | ---------------------------- | --- | --- | ------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | --- | --------------------- | --- | --- |
| | (cid:65)(cid:49)(cid:58)(cid:79)(cid:114)(cid:99)(cid:104)(cid:101)(cid:115)(cid:116)(cid:114)(cid:97)(cid:116)(cid:105)(cid:110)(cid:103) | | | | | | (cid:65)(cid:49)(cid:58)(cid:79)(cid:114)(cid:99)(cid:104)(cid:101)(cid:115)(cid:116)(cid:114)(cid:97)(cid:116)(cid:105)(cid:110)(cid:103) | | GP T 4 .1 | | |
| | | | | | | | chat | @R | es t A P I customer | | |
| chat | | | | | | messages | | | profiles | | |
| messages | | | customer | | | (cid:65)(cid:50)(cid:32)(cid:43)(cid:32)(cid:65)(cid:51)(cid:58)(cid:32)(cid:83)(cid:116)(cid:114)(cid:101)(cid:97)(cid:109)(cid:32) | | | | | |
(cid:65)(cid:50)(cid:58)(cid:32)(cid:83)(cid:116)(cid:114)(cid:101)(cid:97)(cid:109)(cid:32)(cid:105)(cid:110)(cid:103)(cid:101)(cid:115)(cid:116)(cid:105)(cid:111)(cid:110) profiles (cid:105)(cid:110)(cid:103)(cid:101)(cid:115)(cid:116)(cid:105)(cid:111)(cid:110)(cid:32)(cid:43)(cid:32)(cid:70)(cid:105)(cid:108)(cid:116)(cid:101)(cid:114)(cid:105)(cid:110)(cid:103) (cid:65)(cid:53)(cid:58)(cid:32)(cid:67)(cid:117)(cid:115)(cid:116)(cid:111)(cid:109)(cid:101)(cid:114)(cid:32)(cid:101)(cid:120)(cid:116)(cid:114)(cid:97)(cid:99)(cid:116)(cid:105)(cid:111)(cid:110)
(cid:65)(cid:53)(cid:58)(cid:32)(cid:67)(cid:117)(cid:115)(cid:116)(cid:111)(cid:109)(cid:101)(cid:114)(cid:32)(cid:101)(cid:120)(cid:116)(cid:114)(cid:97)(cid:99)(cid:116)(cid:105)(cid:111)(cid:110)
| (cid:65)(cid:51)(cid:58)(cid:32)(cid:70)(cid:105)(cid:108)(cid:116)(cid:101)(cid:114)(cid:105)(cid:110)(cid:103) | | (cid:65)(cid:54)(cid:58)(cid:32)(cid:80)(cid:111)(cid:108)(cid:105)(cid:99)(cid:121)(cid:32)(cid:114)(cid:117)(cid:108)(cid:101)(cid:115)(cid:32)(cid:99)(cid:104)(cid:101)(cid:99)(cid:107) | | | | | | | | | |
| ---------------------------------------------------------------------------------------------------------------- | --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | --- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----------- | --- |
| | | | | | L la m | a 3 .1 (cid:65)(cid:52)(cid:58)(cid:32)(cid:83)(cid:101)(cid:110)(cid:116)(cid:105)(cid:109)(cid:101)(cid:110)(cid:116)(cid:32)(cid:97)(cid:110)(cid:97)(cid:108)(cid:121)(cid:115)(cid:105)(cid:115) | | (cid:65)(cid:54)(cid:58)(cid:32)(cid:80)(cid:111)(cid:108)(cid:105)(cid:99)(cid:121)(cid:32)(cid:114)(cid:117)(cid:108)(cid:101)(cid:115)(cid:32)(cid:99)(cid:104)(cid:101)(cid:99)(cid:107) | I B M G | r a n i t e | |
(cid:65)(cid:52)(cid:58)(cid:32)(cid:83)(cid:101)(cid:110)(cid:116)(cid:105)(cid:109)(cid:101)(cid:110)(cid:116)(cid:32)(cid:97)(cid:110)(cid:97)(cid:108)(cid:121)(cid:115)(cid:105)(cid:115) (cid:64)(cid:73)(cid:110)(cid:102)(cid:101)(cid:114)(cid:101)(cid:110) (cid:99) (cid:101) (cid:32)(cid:115)(cid:101) (cid:114) (cid:118)(cid:101) (cid:114) (cid:64) (cid:73) (cid:110) (cid:102)(cid:101)(cid:114) (cid:101)(cid:110) (cid:99) (cid:101) (cid:32) (cid:115) (cid:101) (cid:114)(cid:118)(cid:101)(cid:114)
| | (cid:65)(cid:55)(cid:58)(cid:32)(cid:69)(cid:110)(cid:116)(cid:105)(cid:116)(cid:105)(cid:101)(cid:115)(cid:32)(cid:105)(cid:110)(cid:116)(cid:101)(cid:103)(cid:114)(cid:97)(cid:116)(cid:105)(cid:111)(cid:110)(cid:32)(cid:38)(cid:32)(cid:109)(cid:101)(cid:114)(cid:103)(cid:101)(cid:32) | | | | | (cid:65)(cid:55)(cid:58)(cid:32)(cid:69)(cid:110)(cid:116)(cid:105)(cid:116)(cid:105)(cid:101)(cid:115)(cid:32)(cid:105)(cid:110)(cid:116)(cid:101)(cid:103)(cid:114)(cid:97)(cid:116)(cid:105)(cid:111)(cid:110)(cid:32)(cid:38)(cid:32)(cid:109)(cid:101)(cid:114)(cid:103)(cid:101)(cid:32) | | | Llama 3.1 | | |
| --- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | --- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | --- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | --- | --- |
| | | | (cid:77)(cid:117)(cid:108)(cid:116)(cid:105)(cid:45)(cid:97)(cid:103)(cid:101)(cid:110)(cid:116)(cid:32)(cid:80)(cid:105)(cid:112)(cid:101)(cid:108)(cid:105)(cid:110)(cid:101)(cid:32) | | | | | | (cid:64)(cid:73)(cid:110)(cid:102)(cid:101)(cid:114)(cid:101)(cid:110)(cid:99)(cid:101)(cid:32)(cid:115)(cid:101)(cid:114)(cid:118)(cid:101)(cid:114) | | |
(cid:79)(cid:112)(cid:116)(cid:105)(cid:109)(cid:105)(cid:122)(cid:97)(cid:116)(cid:105)(cid:111)(cid:110)
| | | (cid:65)(cid:56)(cid:58)(cid:32)(cid:75)(cid:80)(cid:73)(cid:115)(cid:32)(cid:99)(cid:111)(cid:109)(cid:112)(cid:117)(cid:116)(cid:97)(cid:116)(cid:105)(cid:111)(cid:110) | | | | | (cid:65)(cid:56)(cid:58)(cid:32)(cid:75)(cid:80)(cid:73)(cid:115)(cid:32)(cid:99)(cid:111)(cid:109)(cid:112)(cid:117)(cid:116)(cid:97)(cid:116)(cid:105)(cid:111)(cid:110) | | | | |
| --- | --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | --- | --- | --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | --- | --- | --- |
(cid:65)(cid:57)(cid:58)(cid:32)(cid:82)(cid:101)(cid:112)(cid:111)(cid:114)(cid:116)(cid:32)(cid:103)(cid:101)(cid:110)(cid:101)(cid:114)(cid:97)(cid:116)(cid:105)(cid:111)(cid:110)
| | | | | | | | (cid:65)(cid:57)(cid:58)(cid:32)(cid:82)(cid:101)(cid:112)(cid:111)(cid:114)(cid:116)(cid:32)(cid:103)(cid:101)(cid:110)(cid:101)(cid:114)(cid:97)(cid:116)(cid:105)(cid:111)(cid:110) | | Mixtral | | |
| --- | --- | --- | --- | --- | --- | --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | ------- | --- | --- |
(cid:64)(cid:72)(cid:117)(cid:103)(cid:103)(cid:105)(cid:110)(cid:103)(cid:70)(cid:97)(cid:99)(cid:101)(cid:32)(cid:65)(cid:80)(cid:73)
(cid:65)(cid:49)(cid:48)(cid:58)(cid:32)(cid:70)(cid:101)(cid:101)(cid:100)(cid:98)(cid:97)(cid:99)(cid:107)
| | | | | | | | (cid:65)(cid:49)(cid:48)(cid:58)(cid:32)(cid:70)(cid:101)(cid:101)(cid:100)(cid:98)(cid:97)(cid:99)(cid:107) | Claude | | | |
| --- | --- | --- | --- | --- | --- | --- | ------------------------------------------------------------------------------------------------------------ | ------ | --- | --- | --- |
(cid:64)(cid:73)(cid:110)(cid:102)(cid:101)(cid:114)(cid:101)(cid:110)(cid:99)(cid:101)(cid:32)(cid:115)(cid:101)(cid:114)(cid:118)(cid:101)(cid:114)
(cid:65)(cid:98)(cid:115)(cid:116)(cid:114)(cid:97)(cid:99)(cid:116)(cid:32)(cid:77)(cid:117)(cid:108)(cid:116)(cid:105)(cid:45)(cid:65)(cid:103)(cid:101)(cid:110)(cid:116)(cid:32)(cid:80)(cid:105)(cid:112)(cid:101)(cid:108)(cid:105)(cid:110)(cid:101) (cid:79)(cid:112)(cid:116)(cid:105)(cid:109)(cid:105)(cid:122)(cid:101)(cid:100)(cid:32)(cid:69)(cid:120)(cid:101)(cid:99)(cid:117)(cid:116)(cid:97)(cid:98)(cid:108)(cid:101)(cid:32)(cid:77)(cid:117)(cid:108)(cid:116)(cid:105)(cid:45)(cid:65)(cid:103)(cid:101)(cid:110)(cid:116)(cid:32)(cid:80)(cid:105)(cid:112)(cid:101)(cid:108)(cid:105)(cid:110)(cid:101)
Figure1:Initialmulti-agentpipeline(left)andoptimizedmulti-agentpipeline(right).
3 ANALYSISOFMULTI-AGENTPIPELINES
Wenowaskwhetherthisgapissystematic.Weuse62realagentic
| | | | | pipeline | deployments | | spanning | domains | such as | customer | sup- |
| --- | --- | --- | --- | -------- | ----------- | --- | -------- | ------- | ------- | -------- | ---- |
port,financialanalytics,andsupply-chainmonitoring.Wethen
constructandanalyze9,000multi-agentpipelinesbyvaryingtask
compositions.Eachpipelinecomprisesatleastthreetasksspanning
dataingestion,schemamapping,featureengineering,analytical
reasoning,andreporting.Wereportthefollowingkeyfindings.
| | | | | Pipeline | complexity | | is growing. | | Figure 3a shows | the | distri- |
| --- | --- | --- | --- | ---------- | ---------- | --------- | ----------- | ---------- | --------------- | ------- | ------- |
| | | | | bution | of tasks | per | pipeline. | We observe | the highest | density | at |
| | | | | 5–7 tasks, | while | pipelines | with | more | than 10 tasks | are | not un- |
common.Astoolsforautomaticallycreatingpipelinesmature,we
Figure2:Paretofrontierfor10,000randomlysampledplans. expectthisdistributiontoshifttowardevenlargerpipelines,much
asapplication-generatedSQLworkloadshavegrownincomplex-
ity[26].Thistrendmattersbecausetheoptimizationspacegrows
combinatoriallywiththenumberofagents.
| | | | | Modelselectionisdrivenbyhabit,notoptimization. | | | | | | | Across |
| --- | --- | --- | --- | ---------------------------------------------- | --- | --- | --- | --- | --- | --- | ------ |
Optimalplansareheterogeneous. Thecheapestplan($0.01, thecorpus,57%oftasksarestochastic(Figure3b),and21%ofthese
42.5s,quality0.77)assignsLlama3.18Btoallstochasticagentsand useanLLMfordiverseproblems.Yetwhenweexaminewhich
placeseachdeterministicagentonitshomeengine.Thehighest- modelsareactuallyused,weobservethesamelimitedsetrecurring
qualityplan($1.84,125.9s,quality0.95)mixesClaudeSonnet3.5 despitetheexistenceofamuchlargermodellandscape(Figure3d).
andQwen2.5-72Bacrossstochasticagents,deliberatelyusingthe Users strongly prefer models they have already used and tend
cheapermodelwhereClaude’sadvantageismarginal.Abalanced tofavorsmallermodelsoverlargerones.Inotherwords,model
plan($0.64,58.9s,quality0.85)cutscostby65%andlatencyby selectionisnotdrivenbytask-specificcost-qualitytrade-offs,but
53%relativetothehighest-qualityplanbymixingallthreemodels. byfamiliarityandconvenience.Thisreinforcestheconclusionof
Crucially,96%ofPareto-optimalplansusemixedLLMassignments, Section2,thatassigningonefamiliarmodelbroadlyacrossagents
aconfigurationthatusersrarelyconstructmanually. leavessubstantialoptimizationpotentialunrealized.
Topologyasathirdoptimizationaxis. Simpletopologiesdominatedespiteinefficiency.
| | | Thevarianceaboveis | | | | | | | | Themost | |
| --- | --- | ------------------ | --- | --- | --- | --- | --- | --- | --- | ------- | --- |
measuredunderafixedtopology.Toinvestigatethetopology’s commonstructureisasimplechain(34%),followedbyDAGs(25%),
effect as well, we modify only the structure of a balanced plan trees(24%),andbranchingchains(6%)asshowninFigure3e.More
foundintheprevioussetting.Inparticular,wereplacethesingle complexstructuresarerare.Thisdistributionreflectseaseofdevel-
agentA4withaconditionalrefinementloopthatrunsthecheapest opmentratherthanefficiency:chainsexposenoparallelism,and
modelbydefaultandescalatestoClaudeonlywhentheconfidence noneofthedominantstructuresnaturallycapturefeedbackloops.
islow.ThismatchestheaccuracyofrunningClaudeoneverycase Engineheterogeneityisunderexploited. 43%oftasksinvolve
whilepayingtheexpensivemodelpriceonasmallminority,sothe
adeterministicagentrequiringaqueryengine(Figure3c).Most
resultingpipelinelowersbothcostandlatencyatequalquality(cost
oftheserunonstructureddatabases(60%),whiletheremainder
fallsfrom$0.64to$0.44,latencyfrom58.9𝑠to57𝑠).Thisrestructured
aresplitacrossstreamingengines(8%),analyticsengines(23%),
pipelinedominatesthebalancedplanandliesoutsidethePareto andvectordatabases(9%).Thisconfirmsthatengineassignmentis
frontierofthetwo-dimensionalspace,reachingapointnomodel rarelyoptimized.
orengineassignmentcanreachwithoutchangingtopology.
48
assumetheexistenceofaglobalagentregistryR ={𝑎
1
,𝑎
2
,...,𝑎 𝑛}
thatdefinestheavailableagentstogetherwiththeirfunctionalde-
scriptionsandexpectedinputsandoutputs.Formally,wedefine
anabstractmulti-agentpipelineasadirectedgraph𝐺 = (𝐴,𝐸),
where𝐴 ⊆ R is the set of agents participating in the pipeline
and𝐸 ⊆𝐴×𝐴capturesthecommunicationbetweenthem.Each
directededge(𝑎 𝑖 ,𝑎 𝑗) ∈𝐸denotesadependencyinwhichtheout-
(a)Distributionof#tasks. (b)Taskdistribution. putofagent𝑎 𝑖 servesasinputtoagent𝑎 𝑗.Thepipelinetherefore
specifieswhattasksaretobeperformedandtheinformation(i.e.,
controland/ordata)flowsbetweenagents,whileabstractingaway
fromhoweachagentisimplementedorexecutedandhowtheyare
connected.Wedistinguishbetweentwoclassesofagents:stochastic
agents,whoseoutputsmayvaryacrossexecutionsandmaybe
approximate(e.g.,summarization),anddeterministicagents,which
returnthesameoutputforthesameinput(e.g.,databaseretrieval).
Executablemulti-agentpipeline. Givenanabstractpipeline𝐺,
(c)Engineselection. (d)MostpopularLLMfamilies.
anexecutablepipelinespecifies(i)themodelassignedtoeachsto-
chasticagent,(ii)theexecutionengineforeachagent,and(iii)the
structurethatdeterminesexecutionorder,parallelism,andcommu-
nication.Formally,anexecutablepipelineisagraph𝐺∗=(𝐴∗,𝐸∗)
thatmaydifferstructurallyfrom𝐺,where𝐴∗ denotesthesetof
agentswithconcretemodelandengineinstantiationsand𝐸∗shows
thecommunicationamongtheinstantiatedagents.
Transformations.𝐺∗mayhaveadifferentstructurefrom𝐺 be-
(e)Pipelinestopologiesused. causeofastructure-changingtransformation.Weconsiderfour
transformations:(i)fusionmergestwoadjacentagentswhenthe
Figure3:Analysisofmulti-agentdatapipelines. firstfeedsonlythesecond;(ii)splitdecomposesanagentwhenits
sub-agentsreproduceitsinput-outputcontract;(iii)reorder/parallelize
repositionsorparallelizesagentsatanypointwheretheirinputsre-
Redundancyispervasive. Coretaskssuchasdataingestion, mainavailable;andconditionalre-executionaddsaboundedguarded
entityextraction,andreportgenerationrecuracrossthemajority loopthatre-invokesanagentwhileitsqualityestimateislow.In
ofpipelines,yetrepeatedcomputationsarerarelyrecognizedor allcasestheproducer-consumerdependenciesandeachagent’s
shared,leadingtounnecessarycostandlatency. input-outputcontractarepreserved,sothat𝐺∗stayssemantically
consistentwith𝐺.
4 THEOPTIMIZATIONPROBLEM
Problemstatement. Givenanabstractpipeline𝐺 =(𝐴,𝐸)anda
We now formalize the problem of optimizing multi-agent data setofoptimizationobjectives𝑂 =𝑜 1 ,...,𝑜 𝑚 (e.g.,#tokens,mone-
pipelines.Givenanabstractpipeline,thegoalistodeterminean tarycost,quality,runtime,energyconsumption,etc.),thegoalis
executablepipelinebyjointlyselectingpipelinetopology,model tofindoneormoreexecutablepipelines𝐺∗ = (𝐴∗,𝐸∗) thatare
assignments,andexecutionenginesundermultipleobjectivessuch Pareto-optimalwithrespectto𝑂.
as latency, cost, and accuracy. These decisions interact in com- Notethatamongtheobjectivesin𝑂,qualityisdistinctive:unlike
plexandnon-obviousways:Asingleabstractpipelinecanmapto monetarycostorruntime,ithasnouniversalmetricandistask
manystructurallydifferentexecutablepipelines,eachwithdistinct dependent.Wethereforedefinethequalityofanexecutablepipeline
performancecharacteristics. 𝐺∗asatask-specificfunction𝑞(𝐺∗)whichcanbeinstantiatedin
Figure1illustratesthisdistinction.Startingfromanabstract differentways,e.g.,viareference-basedmetrics(precision,recall,F1,
pipeline,agentscanbemerged,split,reordered,orparallelized, exactmatch),model-basedjudges(LLM-as-judge,self-consistency),
whilemodelandengineassignmentsvaryacrossconfigurations. ordomain-specificscores.Theweightedgeometricmeanofper-
Thisresultsinalarge,heterogeneoussearchspace.Toformalize agentaccuracyusedinSection2isonesuchinstantiation.
thissetting,wedefinethefollowingconcepts. Thisproblemformulationshowswhymulti-agentpipelineopti-
Abstractmulti-agentpipeline. Ataconceptuallevel,amulti- mizationcannotbereducedtotraditionalqueryoptimization.First,
agentpipelineisalogicalcompositionofagentscollaboratingto topologyisavariable,ratherthanaplanwithfixedoperators.Sec-
accomplishatask.Eachagentprovidesaspecificcapability,such ond,optimizationisinherentlymulti-objective,requiringtrade-offs
asretrieval,sentimentanalysis,orsummarization,andinteracts acrosslatency,cost,andaccuracy.Third,theabsenceofacommon
withotheragents.Unlikerelationalsystems,wherethespaceof operatoralgebrapreventstheuseofclassicalpruningtechniques.
operationsisfixed,thespaceoftasksthatagentscanperformis Together,thesepropertiesinvalidatethecoreassumptionsunder-
significantlybroaderand,inprinciple,unbounded.Inpractice,how- lyingtraditionaloptimizersandrequireafundamentallydifferent
ever,asystemcanonlyexecuteafinitesetofagents.Wetherefore approach,motivatingthedesignof NOMA.
49
5 VISIONANDRESEARCHCHALLENGES
predictcostonlywithinagivenplanshape,whilecross-engineop-
Optimizingmulti-agentpipelinesrequiresrethinkingeachpillar timizers[1,4,10,17,28]extendthistoheterogeneousenginesbut
ofqueryoptimization.Weidentifythemainchallengesthatarise stillassumeafixedstructure.Second,agentcostsignalsarenotdi-
in this new setting and for each one we describe why existing rectlycomparable:LLMagentsincurtokencosts,stochasticlatency,
approaches are insufficient, present our vision for how NOMA andvariablequality,databaseagentsdependondatavolumeand
addressesit,andidentifytheopenresearchquestionsthatremain. indexing,andstreamingenginesdifferagain.Qualitypropagation
compoundsthis:acheaper,lower-qualityupstreammodelchanges
5.1 VariableMulti-DimensionalSearchSpace thevolumeandnatureofdownstreaminputs,alteringenginecosts
inwaysaper-agentcostmodelcannotcapture.
Thesearchspaceofamulti-agentpipelinespansthreedimensions: Ourvision. Weproposebuildingunifiedcostmodelsthatmaphet-
pipelinetopology,modelselection,andexecutionengineassign-
erogeneousagentfeaturesintoasharedrepresentationandpredict
ment.Thesearehighlyinter-dependent:changingamodelassign-
calibrateddistributionsoverlatency,monetarycost,andaccuracy
mentchangesthedownstreamworkloadthatenginedecisionsmust
ratherthanscalarpointestimates.Ratherthanasinglemonolithic
accountfor,mergingtwoagentseliminatesamodeldecision,while
model,weenvisionafamilyofcostmodels,oneperobjective,that
splittinganagentcreatesnewones.Thesetofdecisionvariablesis
shareacommonpipelinegraphrepresentationbutspecializetheir
thusnotfixedbutchangeswitheachstructuralalternative,yielding
outputheadsforthestatisticalpropertiesofeachobjective.Dis-
avast,variablespacewhereexhaustiveenumerationisinfeasible.
tributionalestimatesareessentialbecauseagentsarestochastic
Besidesthecomplexityofthesearchspace,themulti-objectivena-
and quality propagates non-additively: a point estimate cannot
tureoftheproblemcompoundsthisdifficulty.Thesystemmust
capturetheuncertaintyintroducedbyupstreammodelchoices,
exploreaParetofrontieroverdifferentuser-specifiedobjectives.
makingitimpossibletoreasonreliablyaboutdownstreamcosts.
Thesearchprocessmustcreatediversecandidatesthatreflectdiffer-
Conformalprediction[2,21]canprovidedistribution-freecalibra-
enttrade-offsratherthanconvergingtoasingleoptimalplan.These
tionguarantees.Onlinerecalibration[34]keepsthemodelsaccurate
challengesruleoutclassicalenumeration-basedapproaches,where
aspricing,hardwareavailability,andworkloadsshift.Thethree
algebraic equivalences prune the search space and cost models
objectivesdiffersignificantlyinhowtheycanbelearned.Monetary
estimateasinglecostforthecandidates.
costcanbecomputedalmostanalytically:forstochasticagentsitis
Ourvision. Weproposeusinggenerativemodelstrainedonhis- adeterministicfunctionoftokenvolumeandpublishedper-token
toricalpipelinestogenerateexecutablepipelinesdirectlyinthe pricesandfordeterministicagentsafunctionofdatavolumeand
jointoptimizationspace,bypassingplanenumeration.InNOMA,a resource-timeatknownrates;theonlylearnedcomponentispre-
multi-agentpipelinegeneratorproducescandidatesthatvaryacross dictingvolumebeforeexecution.Latencyislargelyprofiledoffline:
topology,modelassignments,andengineconfigurationssimulta- per-agentlatencydistributionsaremeasuredper(model,engine,
neously,reflectingdifferentpositionsontheParetofrontier.Gen- input-size)andcomposedalongthepipeline,withalearnedcor-
erativemodelshaverecentlybeenproposedforrelationalquery rectionforcontentionwhenagentsshareresources.Qualityisthe
optimization[29],howeverthisworkoperatesoverafixedrela- onlyobjectivethatisfullylearned,label-scarce,andnon-additive,
tionalplanstructureandoptimizesforasingleobjective(execution andisthereforewherethechallengeconcentrates.Contextpassing
time).Similarly,learnedorchestrators[7]optimizeexecutionorder isaccountedforwhenanexecutionplanisevaluated:thecontext
forafixedagentsetunderasingleobjective.Incontrast,NOMA’s anagentreceivesdeterminesitsinputcostandlatencyandaffects
generatormustproducebothpipelinestructureandexecutionde- itsoutputquality.Critically,thecostmodelneednotbeaccurate
cisionsjointly,withoutafixedplanshape,withoutcardinalityes- inanabsolutesense:itonlyneedstorankcandidateswellenough
timates,andacrossamulti-objectiveParetofrontier,makingita toprunegenerationandstaycalibratedtoflagwhenanexecution
fundamentallydifferentgenerationproblem. deviatesfromprediction,triggeringtherefiner.
Openresearchquestions. Traininggenerativemodelsrequires Openresearchquestions. Howcancostmodelscompareconfig-
examplesofgoodpipelines,yetobtainingthemrequiresexecut- urationsacrossstructurallydifferenttopologies?Howshouldhet-
ingmanynon-optimalpipelines:Howcanwebootstraptraining erogeneouscostsignals(tokenusage,datavolume,modelaccuracy)
datawithoutexhaustive,costlyexploration?Whatrepresentations bereconciledintoaunifiedrepresentation?Howcandistributional
capturebothgraphstructureandagentsemanticswellenoughto estimatesstaycalibratedwhenqualitypropagatesnon-additively,
generalizetounseenpipelines?Andhowcangenerationbemade sothatuncertaintyatoneagentaffectsalldownstreamagents?And
cost-aware,sothatcandidatesarelikelyneartheParetofrontier howshouldpipelinequalitybedefinedandestimatedefficiently,
beforethecostmodelevaluatesthem,avoidingwastedevaluations giventhatitistask-dependentandlacksadirectlyobservablelabel?
onpoorcandidates?
5.3 OptimizationUnderStochasticExecution
5.2 UnifiedCostModels
Traditionalqueryoptimizationisaone-shotprocess:theoptimizer
Traditionalandlearnedcostmodelsassumeafixedplanstructure producesaplanbeforeexecutionbeginsandtheexecutorcarries
andproducescalarestimates,assumptionsthatbreakinthemulti- itoutwithoutfurtherintervention.Thisseparationispossiblebe-
agentsetting.First,topologyisavariable:thecostmodelmust causerelationaloperatorsaredeterministicandcostmodelerrors
comparepipelinesdifferinginthenumberofagents,theirconnec- manifestonlyassuboptimalperformance,notasfundamentally
tivity,andtheirexecutionorder.Learnedcostmodels[14,15,28] changeddownstreamworkloads.Multi-agentpipelinesviolateboth
50
conditions.Stochasticagentsintroducenon-determinism:thesame andoptimizationdecisions:modelassignmentsandengineconfig-
inputcanproducedifferentoutputsondifferentruns,andthequal- urationsthatprovedeffectiveinpriorpipelines.Thecacheindexes
ityofanupstreamagent’soutputdirectlyaffectsthevolume,dis- alltheseartifactssemanticallyratherthansyntactically.Itcom-
tribution,andnatureofdownstreaminputs.One-shotoptimization binesstructuralnormalizationwithsemanticembedding,witha
cannotaccountforthisbecausetheactualworkloadisnotknown lightweightcoarsefiltertoeliminateobviousmismatchesbefore
untilexecutionbegins.Existingapproachessuchasprogressive invokingthemoreexpensiveembedding-basedsimilaritycheck.
optimization[22]andadaptivequeryprocessing[9]donotaddress Openresearchquestions. Howshouldsemanticequivalencebe
thissetting:theytriggerre-optimizationreactivelywhenobserved definedforagentinputs,balancingprecision,recall,andsimilarity-
cardinalitiesdeviatefromestimatesandre-planningischeaprela- computationcost?Howcansub-planreusebesupportedacross
tivetoexecution.Inoursetting,revisionmustbecontinuousand structurallydifferentpipelines,wheresimilarityissemanticrather
re-planningisexpensiveenoughthatthesystemmustreasonabout thansyntactic?Andhowshouldstalenessbemanaged,andcached
whethertheexpectedbenefitofrevisionjustifiesitscost. decisionstransferredacrosspipelines?
Our vision. We argue that multi-agent pipeline optimization
shouldbecontinuous.InNOMA,acontinuouspipelinerefinerisa 5.5 NOMAasanIntegratedOptimizationLoop
first-classcomponentthatparticipatesinexecutionfromstartto
finish.ItusesaBayesianoptimizationprocesswhereeachagent Thefourchallengesabovearetightlycoupledfacetsofasingle
completionyieldsanobservationthatupdatescostmodelpriors optimizationloop.NOMA’scontributionisthatitscomponents
andrevisesdownstreamdecisionsaccordingly.Therefinerthen aremutuallydependentandmustbeco-designed.Thegenerator
appliesanacquisitionfunctiontodecidewhethertheexpectedgain proposescandidates,thecostmodelsscorethem,therefinerrevises
fromre-planningjustifiesitscost.BayesQO[31]hasshownthe decisionsatruntime,andthecacheclosestheloopbyconvertingex-
effectivenessofBayesianoptimizationfornavigatingaplanspace ecutionexperienceintoreusableknowledge.Eachofourenvisioned
inrelationalqueryoptimization.Incontrast,NOMA’srefinermust solutionsextendsanexistinglineofworkanddoesnotnecessarily
handlemulti-objectiveoptimizationoveravariablepipelinetopol- constituteastandalonecontribution.Asaconcreteagenda,wepro-
ogyratherthanasingleobjectiveoverafixedquerystructure.The posetoaddressthechallengesinthefollowingorder:(1)aunified
refineristightlycoupledwiththecostmodelsofSection5.2. costmodeloverafixedtopology;(2)pipelinegenerationbuilton
topofit;(3)thecontinuouspipelinerefiner;and(4)caching.
Openresearchquestions. Howshouldthescopeofrevisionbe
determinedaftereachagentcompletion:revisingonlythenext Evaluation. WewillevaluateNOMAonarepresentativesubsetof
decisionischeapbutmyopic,whilerevisingthefullremaining our9,000pipelines,againstbaselinesspanningadeveloper-default
pipelineisexpensivebutgloballyoptimal?Howcanthesystem “habit”configuration,per-agentmodelrouting,orchestration-only
decidewhenrevisionpaysoff,giventhatplanningcostisnolonger approaches,randomsamplingwithParetofiltering,andanoracle
negligible?Howcanstabilitybeensuredunderfrequentrevisions? optimizeronsmallinstancesasanupperbound.Wewillreport
end-to-endcost,latency,andquality.Inaddition,wewillmeasure
5.4 SemanticCachingAcrossPipelines optimizationoverhead,distancetoareferenceParetofrontier,and
component-levelcontributions,suchascachehitrateandrefine-
Multi-agentworkloadsexhibitsignificantredundancy:datainges-
mentgains.WeexpectNOMAtodeliversubstantialreductions
tion,entityextraction,summarization,andreportgenerationrecur
inmonetarycostandlatencyatmatchedqualitylevelsrelativeto
acrosspipelinesandoftenoveroverlappinginputs(seeSection3).
developer-defaultbaselines,whileapproachingtheParetofrontier
InasettingwhereeachLLMexecutionincursnon-trivialmonetary
obtainedbyexhaustiveenumerationonsmallpipelines.
cost,exploitingthisredundancythroughcachingisnotanoptional
enhancementbutacorecostreductionmechanism.Traditional
6 LOOKINGAHEAD
cachingreliesonsyntacticequivalence—acachedresultcanbe
reusedwhenqueriesmatchexactlyorcanbeshownequivalent Multi-agentdatapipelinesrepresentafundamentalshiftinhow
throughalgebraicrewriting.Thisassumptionbreaksdownentirely data-driventasksareexecuted.Yet,theyremainlargelyunopti-
inthemulti-agentsetting.Agentinputsareexpressedinnaturallan- mized.Thechallengesofjointlyoptimizingtopology,modelselec-
guage,executionsarenon-deterministic,andtherearenoalgebraic tion,andexecutionenginesrequirerethinkingeachpillarofquery
rewritingrulestoestablishequivalence.Twoagentinvocationsthat optimization.NOMAisourvisionforaframeworkthataddresses
aresemanticallyequivalentwillnotmatchsyntactically,yetreusing thesechallengesasanintegratedoptimizationloop,whereeach
acachedresultmaybeentirelyappropriate.Semanticcachinghas pipelineexecutionmakesthenextonecheaperandbetter.
beenstudiedsinceDaretal[8]introducedpredicate-basedcaching. Thisdirectionalignswithemergingworkonsynthesizeddata
RecentsystemsadaptittoLLMworkloadsviaembeddingsimilar- systems [12, 19]. NOMA extends this trajectory to multi-agent
ityoverqueries[3,13,33].Thesecacheindividualquery–response workloads:insteadofoptimizingqueriesoverafixedsystem,it
pairs over flat inputs; they do not cache sub-plans or optimiza- enablespipelines,executionstrategies,andsystemconfigurationsto
tiondecisions,reasonaboutstructuralsimilarityacrosspipelines, co-evolve.Inthisview,optimizationisnolongerastageinexecution
ortransferconfigurationstostructurallydifferentpipelines—the butacontinuousfunction,andagenticdatasystemstransitionfrom
challengesNOMA’scachemustaddress. fixedpipelinestoadaptive,self-optimizinginfrastructures.Wehope
Ourvision. Weproposeamulti-purposecachethatgoesbeyond thispaperservesasacalltoactionforthedatabase,systems,and
cachingintermediateresultstoalsostoringoptimizedsub-plans MLcommunitiestotakeonthisproblemtogether.
51
REFERENCES
InternationalConferenceonManagementofData,SIGMODConference2017.ACM,
[1] DivyAgrawal,SanjayChawla,BerttyContreras-Rojas,AhmedK.Elmagarmid, 1717–1722. https://doi.org/10.1145/3035918.3054775
YasserIdris,ZoiKaoudi,SebastianKruse,JiLucas,EssamMansour,Mourad [19] JialeLaoandImmanuelTrummer.2026.GenDB:TheNextGenerationofQuery
Ouzzani,PaoloPapotti,Jorge-ArnulfoQuiané-Ruiz,NanTang,SaravananThiru- Processing-Synthesized,NotEngineered.CoRRabs/2603.02081(2026). https:
muruganathan,andAnisTroudi.2018.RHEEM:EnablingCross-PlatformData //doi.org/10.48550/ARXIV.2603.02081arXiv:2603.02081
Processing-MayTheBigDataBeWithYou!-.Proc.VLDBEndow.11,11(2018), [20] ChunweiLiu,MatthewRusso,MichaelJ.Cafarella,LeiCao,PeterBaileChen,
1414–1427. ZuiChen,MichaelJ.Franklin,TimKraska,SamuelMadden,RanaShahout,and
[2] AnastasiosN.AngelopoulosandStephenBates.2023.ConformalPrediction:A GerardoVitagliano.2025.Palimpzest:OptimizingAI-PoweredAnalyticswith
GentleIntroduction.Found.TrendsMach.Learn.16,4(2023),494–591. https: DeclarativeQueryProcessing.In15thConferenceonInnovativeDataSystems
//doi.org/10.1561/2200000101 Research,CIDR2025.www.cidrdb.org.
[3] FuBang.2023. GPTCache:AnOpen-SourceSemanticCacheforLLMAppli- [21] HanwenLiu,ShashankGiridhara,andIbrahimSabek.2025.ConformalPrediction
cationsEnablingFasterAnswersandCostSavings.InProceedingsofthe3rd forVerifiableLearnedQueryOptimization.Proc.VLDBEndow.18,8(2025),2653–
WorkshopforNaturalLanguageProcessingOpenSourceSoftware(NLP-OSS2023). 2666.
212–218. https://doi.org/10.18653/v1/2023.nlposs-1.24 [22] VolkerMarkl,VijayshankarRaman,DavidE.Simmen,GuyM.Lohman,and
[4] KaustubhBeedkar,BerttyContreras-Rojas,HaralamposGavriilidis,ZoiKaoudi, HamidPirahesh.2004.RobustQueryProcessingthroughProgressiveOptimiza-
VolkerMarkl,RodrigoPardo-Meza,andJorge-ArnulfoQuiané-Ruiz.2023.Apache tion.InProceedingsoftheACMSIGMODInternationalConferenceonManagement
Wayang:AUnifiedDataAnalyticsFramework.SIGMODRec.52,3(2023),30–35. ofData.659–670. https://doi.org/10.1145/1007568.1007642
[5] GoharIrfanChaudhry,EshaChoukse,HaoranQiu,InigoGoiri,RodrigoFonseca, [23] IsaacOng,AmjadAlmahairi,VincentWu,Wei-LinChiang,TianhaoWu,JosephE.
AdamBelay,andRicardoBianchini.2026.Murakkab:Resource-EfficientAgentic Gonzalez,M.WaleedKadous,andIonStoica.2025. RouteLLM:Learningto
WorkflowOrchestrationinCloudPlatforms.In20thUSENIXSymposiumon RouteLLMsfromPreferenceData.InTheThirteenthInternationalConferenceon
OperatingSystemsDesignandImplementation(OSDI26).567–587. LearningRepresentations,ICLR2025.
[6] LingjiaoChen,MateiZaharia,andJamesZou.2024. FrugalGPT:HowtoUse [24] LianaPatel,SiddharthJha,MelissaZ.Pan,HarshitGupta,ParthAsawa,Carlos
LargeLanguageModelsWhileReducingCostandImprovingPerformance.Trans. Guestrin,andMateiZaharia.2025.SemanticOperatorsandTheirOptimization:
Mach.Learn.Res.2024(2024). TowardsAI-BasedDataAnalyticswithAccuracyGuarantees.Proc.VLDBEndow.
[7] YufanDang,ChenQian,XuehengLuo,JingruFan,ZihaoXie,RuijieShi,Weize 18,11(2025),4171–4184. https://doi.org/10.14778/3749646.3749685
Chen,ChengYang,XiaoyinChe,YeTian,XuantangXiong,LeiHan,ZhiyuanLiu, [25] SergeyRedyuk,ZoiKaoudi,SebastianSchelter,andVolkerMarkl.2024.Assisted
andMaosongSun.2025.Multi-AgentCollaborationviaEvolvingOrchestration. designofdatasciencepipelines.VLDBJ.33,4(2024),1129–1153. https://doi.
InAdvancesinNeuralInformationProcessingSystems38:AnnualConferenceon org/10.1007/S00778-024-00835-2
NeuralInformationProcessingSystems2025,NeurIPS2025. [26] TobiasSchmidt,ViktorLeis,PeterBoncz,andThomasNeumann.2025.SQLStorm:
[8] ShaulDar,MichaelJ.Franklin,BjörnÞórJónsson,DiveshSrivastava,andMichael TakingDatabaseBenchmarkingintotheLLMEra. Proc.VLDBEndow.18,11
Tan.1996. SemanticDataCachingandReplacement.InProceedingsof22th (2025),4144–4157. https://doi.org/10.14778/3749646.3749683
InternationalConferenceonVeryLargeDataBases,VLDB’96.330–341. [27] ShreyaShankar,TristanChambers,TarakShah,AdityaG.Parameswaran,and
[9] AmolDeshpande,ZacharyG.Ives,andVijayshankarRaman.2007. Adaptive EugeneWu.2025.DocETL:AgenticQueryRewritingandEvaluationforComplex
QueryProcessing.FoundationsandTrendsinDatabases1,1(2007),1–140. DocumentProcessing. Proc.VLDBEndow.18,9(2025),3035–3048. https:
[10] KaterinaDoka,NikolaosPapailiou,VictorGiannakouris,DimitriosTsoumakos, //doi.org/10.14778/3746405.3746426
andNectariosKoziris.2016. Mix’n’matchmulti-engineanalytics.In2016 [28] AndrásStrausz,NielsPardon,andIoanaGiurgiu.2025.ALearnedCostModel-
IEEEInternationalConferenceonBigData(IEEEBigData2016).194–203. https: basedCross-engineOptimizerforSQLWorkloads.InProceedingsofWorkshops
//doi.org/10.1109/BIGDATA.2016.7840605 atthe51stInternationalConferenceonVeryLargeDataBases.
[11] AnasDorbani,SunnyYasser,JimmyLin,andAmineMhedhbi.2025. Beyond [29] PavelSulimov,ClaudeLehmann,andKurtStockinger.2025.GenJoin:Conditional
Quacking:DeepIntegrationofLanguageModelsandRAGintoDuckDB.Proc. GenerativePlan-to-PlanQueryOptimizerthatLearnsfromSubplanHints.Proc.
VLDBEndow.18,12(2025),5415–5418. https://doi.org/10.14778/3750601.3750685 ACMManag.Data3,4(2025),247:1–247:25. https://doi.org/10.1145/3749165
[12] TimoEckmann,MatthiasJasny,JohannesWehrstein,andCarstenBinnig.2026. [30] XinTan,YiminJiang,YitaoYang,andHongXu.2025. TowardsEnd-to-End
TheFutureIsBespoke:SynthesizingOne-Size-Fits-OneDBMSswithLLMCoding OptimizationofLLM-basedApplicationswithAyo.InProceedingsofthe30thACM
Agents.IEEEDataEng.Bull.50,1(2026),88–103. InternationalConferenceonArchitecturalSupportforProgrammingLanguagesand
[13] WarisGill,MohamedElidrisi,PallaviKalapatapu,AmmarAhmed,AliAnwar, OperatingSystems,ASPLOS2025.1302–1316. https://doi.org/10.1145/3676641.
andMuhammadAliGulzar.2025.MeanCache:User-CentricSemanticCaching 3716278
forLargeLanguageModelBasedWebServices.InProceedingsofthe39thIEEE [31] JeffreyTao,NatalieMaus,HaydnThomasJones,YimengZeng,JacobR.Gard-
InternationalParallelandDistributedProcessingSymposium(IPDPS).IEEE,1298– ner,andRyanMarcus.2025. LearnedOfflineQueryPlanningviaBayesian
1310. https://doi.org/10.1109/IPDPS64566.2025.00117 Optimization. Proc. ACM Manag. Data 3, 3 (2025), 179:1–179:29. https:
[14] RomanHeinrich,XiaoLi,ManishaLuthra,andZoiKaoudi.2025.LearnedCost //doi.org/10.1145/3725316
ModelsforQueryOptimization:FromBatchtoStreamingSystems.Proc.VLDB [32] NoppanatWadlom,JunyiShen,andYaoLu.2026. EfficientLLMServingfor
Endow.18,12(2025),5482–5487. https://doi.org/10.14778/3750601.3750699 AgenticWorkflows:ADataSystemsPerspective.Proc.ACMManag.Data4,3
[15] RomanHeinrich,ManishaLuthra,JohannesWehrstein,HaraldKornmayer,and (2026),169:1–169:29. https://doi.org/10.1145/3802046
CarstenBinnig.2025.HowGoodareLearnedCostModels,Really?Insightsfrom [33] JianxinYan,WangzeNi,LeiChen,XueminLin,PengCheng,ZhanQin,and
QueryOptimizationTasks. Proc.ACMManag.Data3,3(2025),172:1–172:27. KuiRen.2025.ContextCache:Context-AwareSemanticCacheforMulti-Turn
https://doi.org/10.1145/3725309 QueriesinLargeLanguageModels.Proc.VLDBEndow.18,12(2025),5391–5394.
[16] SaehanJoandImmanuelTrummer.2024. ThalamusDB:ApproximateQuery https://doi.org/10.14778/3750601.3750679
ProcessingonMulti-ModalData.Proc.ACMManag.Data2,3(2024),186. https: [34] QihanZhang,ShaolinXie,andIbrahimSabek.2025.LIMAO:AFrameworkfor
//doi.org/10.1145/3654989 LifelongModularLearnedQueryOptimization.Proc.VLDBEndow.18,11(2025),
[17] SebastianKruse,ZoiKaoudi,BerttyContreras-Rojas,SanjayChawla,FelixNau- 4546–4559. https://doi.org/10.14778/3749646.3749712
mann,andJorge-ArnulfoQuiané-Ruiz.2020. RHEEMixinthedatajungle:a [35] YingliZhouandShuWang.2025.TowardstheNextGenerationofAgentSys-
cost-basedoptimizerforcross-platformsystems.VLDBJ.29,6(2020),1287–1310. tems:FromRAGtoAgenticAI[Vision].InProceedingsofWorkshopsatthe51st
[18] ArunKumar,MatthiasBoehm,andJunYang.2017.DataManagementinMachine InternationalConferenceonVeryLargeDataBases.
Learning:Challenges,Techniques,andSystems.InProceedingsofthe2017ACM
52