← Back to summary
Beyond Accuracy: Disentangling Decomposition from Typed-State Quality Assurance in Multi-Agent LLM Pipelines
Converted from PDF with markitdown. Layout may be rough.
Beyond Accuracy: Disentangling Decomposition from Typed-State Quality Assurance in Multi-Agent LLM Pipelines MariaGabrielaAlvesZuppardo ViniciusCardosoGarcia CentrodeInformática,UniversidadeFederalde CentrodeInformática,UniversidadeFederalde Pernambuco(UFPE) Pernambuco(UFPE) Recife,PE,Brazil Recife,PE,Brazil mgaz@cin.ufpe.br vcg@cin.ufpe.br ABSTRACT Noneofthestudieswesurveyedisolatesthecontributionof Multi-agentLLMpipelinesareoftencreditedwithgainsinsoft- anarchitecturalmechanism,suchasatypedinter-agentcontract, wareengineeringtasks,butitisrarelyclearwhetheragainstems fromthatofpromptdesign.Whenapipelineoutperformsasingle- fromthetypedcontractbetweenagentsorfromthespecialized agentbaseline,thegainisoftenreadasevidencethatthecontract prompts that decomposition enables. We study this attribution improvesoutputquality;whetherpromptingalonewouldproduce probleminRequirementsEngineering,comparingthreearchitec- thesamegainisrarelychecked,andsystematicreviewsconfirm turesforclassifyingrequirementsasfunctionalornon-functional thatthisconflationremainsunresolved[6,18](§3).F1isnotthis andthenprioritizingthemwithMoSCoW:asingle-callbaseline;a paper’sobjectofinterestbuttheinstrumentweusetoruleout two-callvariantthatreusesthepipeline’sspecializedpromptsbut oneexplanationbeforeaskingamorespecificquestion:weshow passesanunvalidatedintermediate;andapipelinewithaPydantic- thatthetypedcontractbarelymovesF1(Δ s ≈0),whichpointsto validatedtypedstate.The18conditionscoverPROMISENFR+(625 decompositionanditstask-specificpromptsastheaccuracydriver requirements), three locally hosted small language models, and andleavesopenwhatdifferencesaccompanythetypedcontract,a twolanguages.TheF1gainoverthebaselineisassociatedwith questionF1cannotanswer(§6.5,§6.6). thetwo-call,specialized-promptvariant(upto+0.075),whereas Wethereforeasknotonlywhetheramulti-agentpipelinere- thetypedstateaddsasmalldifference(−0.003to+0.013)thatis finesclassificationaccuracy,butwhicharchitecturalelementdrives significantinoneofsixcomparisons.Beyondaccuracy,thetyped anygainandwhatdifferencesaccompanythetypedcontract.We pipelinecontainsout-of-rangemodeloutputsattheagentbound- operationalisequalityassurancenarrowly,asearlystructuralval- ary,whiletheunvalidatedvariantfailslateranddropstherequire- idationandcross-modelbehaviouralconsistency,looselyrelated ment,anditshowsasmall,reproducibleincreaseininter-model toISO/IEC25010’sReliability(wedonotmeasureMaintainability) MoSCoWconsistencywhosecauseourdesigncannotisolate.The ratherthantheFunctionalSuitabilitythatF1alreadycaptures.We studyisassociative:thecontrastsbundleprompts,callcount,and comparethreearchitecturesonFR/NFRclassificationfollowedby stagedcontext,andtheresultscomefromthree7–8Bmodelsonone MoSCoWprioritisation,usingPROMISENFR+[15,16](625require- dataset.Theimplementationandreplicationpackagearearchived ments):abaselinethatissuesasingleLLMcallperrequirement;a onZenodo(https://doi.org/10.5281/zenodo.23380444). pipelineoftwospecialisedagentsorchestratedwithLangGraph[9] overaPydantic-validatedtypedstate;andatwo-callbaselinethat KEYWORDS reusesthepipeline’stask-specificpromptsbutpassesanunvali- datedintermediateobject.Becausethesecondcallalsoreceives SoftwareQualityAssurance,LLMAgents,Multi-AgentSystems, thefirstcall’sdecidedtypeandcategory,thebaseline-to-two-call RequirementsEngineering,SchemaValidation,EmpiricalStudy, differenceisadecompositioneffectthatbundlespromptwording, MoSCoWPrioritisation,PROMISENFR+ call count, and staged context (§4.5, §6.5). All three run on the 1 Introduction samethreelocallyservedSLMsanddecodingsettings,yieldingan 18-conditiongrid(3models×3architectures×2languages). LargeLanguageModels(LLMs)areincreasinglydeployedascompo- Here,“disentangling”referstocomparingarchitecturalcontrasts; nentsoflargersoftwarepipelines,wheretheiroutputsfeeddown- itdoesnotimplycausalisolationofthetypedcontract’seffect. stream decisions with limited human review [10]. This raises a Contributions.(1)Acontrolledempiricalstudy(18conditions, qualityassurancequestionthathasreceivedlittlecontrolledempir- 𝑛=625;pairedMcNemarandWilcoxontests,Cohen’sℎandpaired icalattentionintheBrazilianSoftwareQualitycommunity:when 𝑔, Bonferroni; Fleiss’𝜅 with a paired bootstrap; and a sensitiv- apipelineisdecomposedintospecialisedagentsconnectedbya ity analysis retaining parse-failure fallbacks) that separates the typedcontract,whatdoesthatcontractbuybeyondtheaccuracyof accuracyeffectofdecompositionfromthequality-assuranceprop- theunderlyingmodelcalls?RecentSBQSworksharestheevaluate- ertiesthataccompanyatypedcontract,corroboratedbyadirect beyond-accuracy concern in adjacent settings, such as judging baseline-vs-two-callcomparison(§6.5).(2)Evidencethatanear- LLM-generatedunittestsagainstmutation-testingcriteria[1]and zeroF1differencedoesnotmakethetypedcontractdispensable, assessinggenerativeAIasasubstituteforusabilityinspectors[2]. restrictedtotwoproperties:containmentofout-of-rangefieldsat Practitionerslikewisecitereliabilityandweakcontextualground- theagentboundary(classificationfaultsonecallearlierthanthe ingasprimarybarrierstoadoptingLLMsinRequirementsEngi- unvalidatedablation),verifiedbyfaultinjection(§6.6),andasmall, neering[14]. 6202 tcO 22 ]ES.sc[ 1v08213.0162:viXra SBQS26,Nov10–13,2026,Maringá,PR Zuppardoetal. statisticallyconfirmedincreaseininter-modelMoSCoWconsis- systemsaregainingtractioninsoftwareengineeringtasks,driven tency(§6.3).Confidence-awarerouting,theothermechanismthe bythedecompositionofworkamongspecializedagentsandex- designattributestothetypedstate,firedforonly3of3,750pipeline plicitorchestrationmechanisms[10].Althoughexistingstudies predictions (all parse fallbacks), so it is validated only through oftenreportend-to-endperformancegains,fewisolatetheeffects thesyntheticfault-injectiontest,notonthePROMISENFR+grid. ofdecompositionandthearchitecturaldecisionsgoverningagent (3)Anopen-sourcepipelinebuiltonareusableBaseAgent[Item, interaction.Thisworkaddressesthisattributionproblemthrougha Output]abstraction.(4)Areplicationpackage(prompts,runner, controlledexperimentaldesign.Wecomparethreeconfigurations: manifests,rawpredictionsforall36executions,analysisscripts, asingle-callbaseline,amulti-agentpipelinewithtypedcontracts tests)archivedonZenodo(DOI10.5281/zenodo.23380444). betweenagents,andatwo-callbaselinewithouttypedstate.By keepingtheunderlyingmodelsanddecodingconfigurationscon- 2 Background stant, the design makes it possible to distinguish the effects of LLM-based agents. LLMs have evolved from passive text gen- task-specificdecompositionandpromptingfromthoseassociated erators into agents that invoke tools, maintain state, and work withthetypedcontractbetweenagents.Thegapweclaimisrelative withotheragents[10].Multi-agentsystemscomposesuchagents tothetwosystematicreviewsaboveandtoSBQS/SBESproceedings intopipelinesinwhicheachhandlesasub-taskandcommunicates of2024–2025;oursurveywasnotsystematic. through typed messages or shared state, reflecting classical de- 4 ImplementationandExperimentalPlatform signprinciples(taskspecialisation,typedcontracts,composability). Theirempiricalbenefitremainsopen,sincesharedcontextmayoff- AllthreearchitecturesaddressthesamecompoundREtask,FR/NFR setthegainsofdecomposition,anditsplitsintotwofacetsthatthis classificationfollowedbyMoSCoWprioritisation,andsharethe papertreatsseparately:functionalaccuracy,andqualityattributes datasetadapter,modelfactory,decodingparameters,retrypolicy, (e.g.,ISO/IEC25010Reliability)thatanaccuracymetricdoesnot concurrencylevel,andmetriccomputation.AsTable1shows,they capture. differinmorethantheintermediaterepresentation;westateeach REandNFRclassification.PROMISENFR+[15,16]provides differencesothateffectsarenotattributedtoasinglefactor(§4.5). 625requirementsacrossonefunctionalclassandtenNFRcate- gories,thecanonicaldatasetforcontrolledclassificationstudiesin 4.1 ArchitecturalOverview theBraziliancommunity[11].MoSCoWprioritisation[3](Must, The baseline classifies and prioritises in a single LLM call Should,Could,Won’tHave)lacksapublicgroundtruth,which (BaselineAgent, joint prompt, no intermediate state). The motivatestreatingMoSCoWconsistencyasaqualityproxyrather two-call baseline (TwoCallBaselineAgent) uses the same thananaccuracymetric(§6.3). task-specific prompt builders as the pipeline but passes the Smalllanguagemodels.SLMs(≤8Bparameters)arecompeti- intermediateresultinanunvalidatedPythondataclass,soPydantic tiveonSEclassificationtasks[13,19],andlocaldeploymentenables validationhappensonlywhenthefinaloutputisassembled.The controlled,versionedexperiments.Structured-outputcompliance, pipelinedividestheworkbetweentwoagentsthatexchangea however,variesacrossSLMs[7],afailuremodethatmattersinside validated Pydantic object, orchestrated by LangGraph. In both pipelines,whereamalformedoutputatoneagent’sboundarycan two-call arms the prioritisation prompt receives the type and spreadintothenextagent’scontext(§6.6). categorydecidedbythefirstcall. Comparingthebaselinewiththetwo-callbaselinebundlesde- 3 RelatedWork composition,callcount,task-specificprompts,andstagedcontext. RecentworkhasinvestigatedtheuseofLargeLanguageModels Comparing the two-call baseline with the pipeline bundles the (LLMs)inRequirementsEngineering(RE)fortaskssuchastheclas- typedcontract,thevalidationboundary,thehandlingofconfidence, sificationoffunctionalandnon-functionalrequirements(FR/NFR), andLangGraphorchestration.Neithercomparisonisolatesasingle prioritization,traceability,andrequirementsgeneration.However, component. moststudiesevaluatethemodelsasstandalonepredictorsandem- phasizeclassificationaccuracy,payinglessattentiontothearchi- 4.2 DesignandPipeline tecturalmechanismsthatcoordinatemultipleinteractionsbetween Thecodefollowsalayeredstructurewithaone-waydependency LLMs.FR/NFRclassificationhasbeenextensivelyinvestigatedus- rule(presentation,application,domain,infrastructure;thedomain ingthePROMISENFR+benchmark[15,16].Recentstudiescom- layerimportsfromnoneoftheothers).Fourpatternscarrythe pareproprietaryandopen-sourcemodelsundervariousprompting experimentalcontrol:anAbstractFactory(build_llm)selectsthe strategies[11],combinefine-tunedSmallLanguageModels(SLMs) providerbymodel-nameprefix;anAdapter(PromiseAdapter)iso- withexplainabilitytechniques[13],andanalyzeperformancedif- lates the dataset schema from the prompts; a Template Method ferencesbetweencompactandproprietarymodels[19].Ingeneral, (BaseAgent[Item, Output])fixestheagentcontract;andaStrat- theseworksaimtoimprovepredictiveperformancewhilekeeping egy(OrchestrationStrategy)letsonerunnerexecuteeveryar- theunderlyingprocessingarchitectureunchanged.Fromabroader chitecturefromthesameconfiguration.Promptsareversionedfiles perspective,systematicreviewsbyZadenoorietal.[18]andHem- whoseversionisrecordedineachmanifest. matetal.[6]identifygrowinginterestintheuseofLLMsinRE The pipeline consists of ClassificationAgent and butalsopointoutalackofrigorous,reproducibleevaluationsofar- PrioritizationAgent. Agents process requirements in a chitecturaldesigndecisions.Concurrently,LLM-basedmulti-agent pool of three workers, retry each call up to three times with BeyondAccuracy:Decompositionvs.Typed-StateQA SBQS26,Nov10–13,2026,Maringá,PR Table1:Differencesbetweenthethreearchitectures.Sharedbyall:model,temperature,workers,retrypolicy,datasetadapter, responseparsing,andmetrics.ENconditionsusetheoriginalEnglishtextandanEnglishprompt;PTconditionsusethe translatedtextandaPortugueseprompt. Property Baseline Two-call Pipeline LLMcallsperrequirement 1 2 2 Prompts joint task-specific task-specific(sameastwo-call) Intermediateresult none plaindataclass validatedPydanticobject FirstPydanticvalidation afterthesinglecall afterthesecondcall afterthefirstcall(agentboundary) Confidencegiventotheprioritiser — fixedat1.0 passed;below0.70thepromptasksforcon- servativelabels Orchestration directagentcall directagentcall LangGraphStateGraph exponential back-off, and preserve input order; the global rank retainedonlyforthetwo-callsupplementruns.Themaingridhas isrecomputedbysortingonpriority_score.Theprioritisation 12conditions:sixforthebaselineandsixforthepipeline(three promptreceivestheclassifier’sconfidenceand,below0.70,instructs models,twolanguages);thetwo-callablationaddssix,for18in themodeltobeconservative;thethresholdisadesignchoice,not total. acalibratedparameter.TheworkflowisaLangGraphStateGraph Theunittestscoveragentparsing,categorynormalisationand with the fixed topology classifier → prioritizer → END; fallbacks, the domain models, the runner and manifest, the sta- onlythepipelineusesLangGraph. tisticalroutines,thelanguagerule(atestfailsifanyagentsends The agents share a PipelineState (a Pydantic PortuguesetextinanEnglishcondition),andtheconfidencerule BaseModel) whose requirement entity grows by inher- (thepipeline’sprioritiserreceivestheclassifier’sconfidence;the itance (Requirement → ClassifiedRequirement → two-call baseline fixes it at 1.0). A fault-injection campaign in- PrioritizedRequirement).Agentoutputshaveboundedfields: jects13faults(sixvalue-rangeviolations,e.g.confidence1.4or confidenceandpriority_scoreliein[0,1],priority_rankis priority_score−0.1,andsevenparse-levelfaults);thetestsassert atleast1,andtherequirementtypeandMoSCoWlabelareenumer- thatthetypedpipelinecontainseachvalue-rangeviolationatthe ations.AvalueoutsidetheseboundsraisesaValidationError agentboundary,whereastheunvalidatedintermediatefailsonly whenthetypedobjectisbuilt,insidetheagentandimmediately whentheoutputmodelisassembled(§6.6). afterthecall;validationisthusapropertyoftheseconstructors, notofLangGraph’sstateassignment.Theagent’sparsercatches 4.5 KnownLimitationsoftheImplementation theerrorandsubstitutesaflaggedfallback(§4.3). Five aspects bound what the experiments can attribute. First, confidence-aware routing is wired but almost never fires: only 4.3 RobustnessLayer threeof3,750pipelinepredictions(twoforqwen2.5:7bPT,onefor Smallmodelsfrequentlyreturnmalformedoroff-schemaJSON. mistral:7bEN,allparsefallbacksexcludedfromthemetrics)have Every agent extracts the outermost JSON object with a regular confidencebelow0.70,soitsisolatedeffectcannotbeevaluated; expression(duplicatedineachagentratherthancentralised).The thetwo-callbaselinefixestheconfidenceat1.0.Second,response classification-producingagentsadditionallynormalisethecategory parsingisregex-basedandduplicatedacrossagents,soaresponse (mappingPortugueseorEnglishprosetothePROMISEcode)and withseveralbrace-delimitedblocksfailsextractionandbecomes repairarecurrentoff-schemapatterninwhichthemodelreturns afallback.Third,seedgovernsdatasetsamplingonlyandisnot anNFRcategorycode(e.g.,PE)intherequirement-typefield,read- forwardedtothemodelserver,sodeterminismrestsontempera- ingitasNF.Therepairwasneededonlybymistral:7b,in16 ture0.Fourth,agentswritetheirresultsintothesharedstateobject of625PTand7of625ENoutputs,withidenticalcountsinthe inplace.Fifth,weevaluatedthedesignonlyinRE,althoughwe pipelineclassifierandthetwo-callbaseline.Ifparsingorvalida- expecttheprinciplestoapplytoothermulti-stageSEtasks. tionstillfails,theagentreturnsadeterministicfallbackinstead ofraising:FUNCTIONALwithconfidencezeroforclassificationand 5 Methodology COULD_HAVEwithscore0.5forprioritisation,soabatchisnever 5.1 ResearchQuestions aborted.Fallbacksareidentifiable(confidencezeroandanerror Two research questions guide this study, complemented by a string)andaretheoutputsexcludedorretainedinthesensitivity language-effectanalysis(§6.2)thatisreportedalongsidetheRQs analysis(§5.6). butisnotitselfaformalRQ,sinceitvariestextlanguageandprompt 4.4 Execution,Traceability,andVerification languagejointlyratherthanisolatingeitherasacontrolledfactor. RQ1.Towhatextentdoesthepipelineoutperformthesingle- OneExperimentRunnerexecuteseveryarchitecturefromafrozen agentbaselineinFR/NFRbinaryclassification?WereportMacro configuration and writes a manifest (strategy, model, language, F1andtestthepairedper-requirementoutcomesacrossthesix seed,temperature,promptversion,datasetpathandMD5hash, model–languagecombinationswiththeWilcoxonsigned-ranktest LangGraphversion,elapsedtime,codecommit)andaresultsfile (confidence-zerofallbacksexcluded)andtheexactMcNemartest withthemetricsandeveryprediction.Per-callJSONLtraceswere SBQS26,Nov10–13,2026,Maringá,PR Zuppardoetal. (fallbacks retained), with Bonferroni correction (𝛼′ = 0.05/6 = 5.4 ModelsandConfiguration 0.0083). WeevaluatethreeSLMsservedlocallyviaOllama0.24.0(Table2). RQ2(central).Towhatextentdoesdecompositionwithitstask- Local deployment removes network and API dependencies, al- specificprompts,ratherthanthetypedinter-agentstatecontract, thoughdecodingattemperature0isnotbit-deterministic(§7).Prior accountfortheF1differencebetweenthepipelineandthesingle- workreportsthatSLMsachieveapproximately2pplowerF1than agentbaseline,andwhatdifferencesaccompanythetypedstate largeproprietarymodelsonPROMISE-familydatasets[13,19]. contractthatF1doesnotcapture?Weexaminethisthroughtwo architecturalcontrasts,Δ p =F1(two-call)−F1(baseline)andΔ s = Table2:Modelsevaluated.AlluseQ4_K_Mquantization. F1(pipeline)−F1(two-call).Thefirstbundlestask-specificprompts, anadditionalcall,andstagedcontext.Thesecondbundlesthetyped Model Params Quant. Pydantic state, its validation boundary, confidence-conditioned prompting(whichfiredforonly3of3,750pipelinepredictions, qwen2.5:7b 7.6B Q4_K_M §4.5), and LangGraph orchestration. Neither contrast isolates a llama3.1:8b 8.0B Q4_K_M singlemechanism,andbecausetheunifiedbaselinepromptand mistral:7b 7.2B Q4_K_M thetask-specificpromptscannotbemadestrictlyidentical(§7),Δ s isreadasanupperboundontheF1differenceassociatedwiththe Fixedhyperparameters:temperature=0.0(greedydecoding); typedstateratherthanasapointestimate.Validationanderror- seed= 42fixesthedatasetsampleonlyandisnotforwardedto propagationbehaviourareexaminedseparately(§6.5,§6.6). themodelserver.Allexperimentswererunonasinglemachine Inter-modelMoSCoWconsistencyisreportedasasoftwarequal- (Intel Core i5-13450HX, 16 GB RAM, NVIDIA RTX 3050 6 GB). ityobservation(Fleiss’𝜅withapairedbootstrapofΔ𝜅;§6.3)rather Each agent processes requirements in parallel (max_workers = thanasaccuracy,becausePROMISENFR+providesnopriority 3)withexponential-backoffretryuptothreeattempts(30–240,s, groundtruth. tenacity). Complementaryanalysis:languageeffect.Wecomparean Englishcondition(originaltextandEnglishprompt)withaPor- 5.5 ExperimentalDesign tuguesecondition(translatedtextandPortugueseprompt)across Thegridcomprises18conditions:3models×3architectures×2lan- allthreearchitectures.Becausetextandpromptlanguagechange guages(Table3).Allconditionsusethecompletedataset(𝑛=625, together,theresultscapturetheirjointeffectratherthaneitherin temperature=0.0);seed42onlyfixesthedataset-samplingroutine, isolation;theresidualconfoundisacknowledgedin§7(§6.2). whichhereselectseveryrequirement.Conditionsrunsequentially 5.2 Dataset onasinglemachinetoavoidGPUcontention,andtherunnersup- portsresumptionvia–resume.Thereplicationpackage(prompts, WeusePROMISENFR+[15,16],thecanonicalbenchmarkforre- manifests,rawpredictions,andanalysisscripts)isarchivedonZen- quirementsclassification,comprising625requirements:255func- odo(https://doi.org/10.5281/zenodo.23380444);thedatasetisnot tional(F)and370non-functional(NF)across10categories(A,FT, redistributed,anditsfilechecksumisrecordedineverymanifest LF,MN,O,PE,PO,SC,SE,US). andinthepackagenotes. TheoriginaldatasetisinEnglish.Tosupportthelanguage-effect analysis,weproducedaPortuguesevariantbymachine-translating Table3:Experimentalgrid(18conditions). eachrequirementwithdeep_translator(GoogleTranslate,2024). Bothvariantsshareidenticalground-truthlabels.Wecheckedtrans- lationqualityviaback-translation(PT→EN,BERTScoreF1with Factor Levels roberta-large):themeanscorewas0.9646acrossall625pairs, Architecture baseline,pipeline,two-callbaseline and624of625scored≥0.90.Thesinglebelow-thresholditemwas Model qwen2.5:7b,llama3.1:8b,mistral:7b manuallyinspectedandconfirmedsemanticallyequivalent. Language EN(original),PT(machine-translated) 5.3 ArchitecturalVariants 𝑛=625,temperature=0.0,seed=42 Thearchitecturalvariantistheprimaryindependentvariablefor RQ1:baseline,pipeline,andtwo-callbaseline(§4)allacceptthe 5.6 EvaluationMetricsandStatisticalAnalysis same625requirementsandproduceoutputsofidenticalstructure, Classification (RQ1, language effect). The primary outcome sothatarchitectureisthemainmanipulatedfactor(theremaining isbinaryFR/NFRclassificationagainstPROMISENFR+labels,re- implementationdifferencesarelistedinTable1and§4.5).Their portedasMacroF1(primary;itweightstheFandNFclassesequally mechanicsaredescribedin§4.1–4.2;inbrief,thebaselineissues despitethe255vs.370imbalance),Accuracy,andtheMatthewsCor- oneunifiedLLMcall,thepipelinedecomposesthetaskviaatyped relationCoefficient(MCC).NFRsubcategoryF1isdiagnosticand PipelineState, and the two-call baseline reuses the pipeline’s computedovertheground-truthNFrequirementsalsoclassified promptswithoutthetypedcontract. correctlyasNF;subcategoryprecisioniscomputedagainstevery emissionofthatlabelinthecondition.Categorieswith𝑛<30(PO 𝑛 =1,FT𝑛 =10,MN𝑛 =17,A𝑛 =21,SC𝑛 =21)areexcluded fromtheprimaryconclusions(§6.4). BeyondAccuracy:Decompositionvs.Typed-StateQA SBQS26,Nov10–13,2026,Maringá,PR | MoSCoWconsistency(qualityobservation).Asnopriority | | | | | 6 Results | | | | | | | -------------------------------------------------- | --- | --- | --- | --- | --------- | --- | --- | --- | --- | --- | groundtruthexists,wemeasureinter-modelagreementwithFleiss’ WereportRQ1(§6.1),thelanguage-effectanalysis(§6.2),MoSCoW 𝜅 [5], treating the three models as raters, within each of the 6 consistency(§6.3),NFRsubcategorydiagnostics(§6.4),RQ2(§6.5), (architecture×language)strata(𝑛 =623inall;the2dropped andvalidationtiming(§6.6). aligned textsarethesamealignmentexclusionusedforRQ1),interpreted Inshort,mostofthepipeline’sF1gainisassociatedwiththe | with the Landis–Koch | | scale [8]. | To test whether | the pipeline’s | | | | | | | | -------------------- | --- | ---------- | --------------- | -------------- | --- | --- | --- | --- | --- | --- | two-callvariantwithspecialisedpromptsratherthanwiththetyped incrementoverthetwo-callbaselineisdistinguishablefromzero, state,whichisaccompaniedbyearliercontainmentofout-of-range werunapairedpercentilebootstrap(𝐵=10,000,sameresampled valuesandasmall,unisolatedconsistencydifference. | | | | Δ𝜅 𝜅 | −𝜅 | | | | | | | | --- | --- | --- | ---- | --- | --- | --- | --- | --- | --- | --- | indicesforbotharchitectures)on = ,with Uniform exclusion policy. All accuracy, F1, and MCC fig- | | | | pipeline | two-call | | | | | | | | ------------------------ | --- | --- | -------------------- | --------- | --- | --- | --- | --- | --- | --- | | Bonferronicorrectionover | | 𝑁 | = 2languagestrata(𝛼′ | = 0.025). | | | | | | | uresarecomputedafterexcludingconfidence-zeroparse-failure Becausethebootstrapresamplesrequirementsbutnotexecutions, fallbacks(38outputsacross4ofthe18conditions),whicharein- wealsore-executedthetwelvepipelineandtwo-callconditions frastructureartefactsofthesharedparsinglayerratherthanmodel onceandcompareΔ𝜅acrossthetwoexecutionsandagainstthe𝜅 predictions,appliedidenticallytoallconditions(§6.6detailswhich variationbetweenrepeatedruns(§6.3). areaffected).Becauseexclusionisananalyst’schoice,wealsoreport RQ1pairedcomparison.Foreachofthe6(model×language) thereadingthatretainsthemasincorrectpredictions:McNemar pairs,theexactMcNemartest[12]onthediscordantpairscom- testsandCohen’s𝑔arecomputedonall623alignedpairs,andthe | pares per-requirement | | outcomes | of pipeline | and baseline over | | | | | | | | --------------------- | --- | -------- | ----------- | ----------------- | --- | --- | --- | --- | --- | --- | Wilcoxonsensitivityisstatedwhereveritchangesaconclusion.Ex- 𝑛 =623alignedrequirements(joinedbyrequirementtext,since clusionistheprimaryanalysisbecauseitisolatesmodelbehaviour; Requirement.id is regenerated on every load; two texts occur theretainedreadingistheend-to-endsensitivityanalysis,inwhich twiceverbatiminPROMISENFR+andcollapsetoonekeyeach,so aparsingfailureisanunsuccessfulsystemoutcome,anditisre- 𝑛=623;underexclusionapairalsoneedsanon-fallbackprediction portedinfullbesideit.Theconclusionsof5ofthe6RQ1conditions onbothsides,givingeffective𝑛of621,623,623,590,623,and622for andof8ofthe9language-effectcellsareunaffected;theexceptions thesixconditions).Becausetheoutcomesarebinary,theWilcoxon arediscussedin§6.1–§6.5. signed-ranktest[17]onthesamepaireddifferencesreducestothe samediscordant-paircounts;itistheprimarytestonthefallback- 6.1 RQ1:Pipelinevs.BaselineClassification excludeddata,andwereportboth.Bothtestper-requirementcor- Thepipelineachievesstatisticallysignificantimprovementsin3of rectness,notMacro-F1,sowealsotesttheMacro-F1differencesdi- 6conditionsafterBonferronicorrectionundertheprimary(exclu- rectlywithapairedbootstrapresamplingrequirementswithineach sion)analysis,andin4of6underthepairedMcNemartestthat | trueclass(𝐵=10,000;Bonferroni-adjustedintervals,𝛼′ | | | | =0.05/6, | | | | | | | | -------------------------------------------------- | --- | --- | --- | -------- | --- | --- | --- | --- | --- | --- | retainsthefallbacks(Table4).Thesetestscompareper-requirement overthe6comparisonspercontrast).TheeffectsizeisCohen’s | | | | | | correctness; | a paired, | class-stratified | bootstrap | of | the macro-F1 | | ----------------- | ---------------- | --- | -------- | ------------------ | ------------ | --------- | ---------------- | --------- | --- | ------------ | | ℎ [4] (thresholds | 0.20/0.50/0.80). | | Becauseℎ | uses only marginal | | | | | | | difference(Table5)givesthesameverdictunderexclusion:the accuracies,wealsoreportthepaired𝑔=𝑏/(𝑏+𝑐)−0.5onthe𝑏+𝑐 | | | | | | Bonferroni-adjusted | interval | excludes | zero | for qwen2.5:7b | PT, | | --- | --- | --- | --- | --- | ------------------- | -------- | -------- | ---- | -------------- | --- | discordantpairswithanexact(Clopper–Pearson)95%intervaland | | | | | | llama3.1:8bPT,andmistral:7bEN,andforΔ | | | | inthefirsttwo. | | | --- | --- | --- | --- | --- | ------------------------------------- | --- | --- | --- | -------------- | --- | thresholds0.05/0.15/0.25;amagnitudeisestimableonlyifitsinter- p Δ | | | | | | For s it | excludes zero | only for | mistral:7b | EN . | The fourth is | | --- | --- | --- | --- | --- | -------- | ------------- | -------- | ---------- | ---- | ------------- | vallieswithinoneband.Bonferronicorrectionover6comparisons llama3.1:8bEN,non-significantunderexclusion(𝑝 =0.680)but gives𝛼′ =0.05/6=0.0083. | | | | | | significantunderMcNemar(𝑝 | | =0.0073)andunderWilcoxonwith | | | | | --- | --- | --- | --- | --- | ------------------------- | --- | ---------------------------- | --- | --- | --- | Languageeffect.Foreachofthe9(architecture×model)pairs, | | | | | | fallbacks | retained (𝑝 | = 0.0055): | the 33 fallbacks | | removed from | | --- | --- | --- | --- | --- | --------- | ----------- | ---------- | ---------------- | --- | ------------ | PTandENoutcomesarealignedbyrequirementtextandcompared | | | | | | the baseline | in that cell | hid the | gap. Cohen’sℎ | places | every ef- | | ------------------------- | --- | --- | -------------------------- | --- | ---------------------------- | ------------ | ------- | ----------------------------- | ------ | --------- | | withtheexactMcNemartest(𝑛 | | | =623;𝛼′ =0.05/9=0.0056).An | | | | | | | | | | | | | | fectinthenegligiblerange(|ℎ| | | | <0.20),butℎignoresthepairing. | | | earlierunpairedanalysis(Mann–WhitneyU)isinappropriate,since Thepaired𝑔islargeforqwen2.5:7bPTandllama3.1:8bPTand bothlanguagesclassifythesamerequirements,andisreportedonly | | | | | | medium for | llama3.1:8b | EN | and mistral:7b | EN, | and its 95% | | --- | --- | --- | --- | --- | ---------- | ----------- | --- | -------------- | --- | ----------- | forcomparison.Uncorrected𝑝-valuesarereportedthroughout. intervalstayswithinonemagnitudebandonlyforthefirsttwo Parse-failuresensitivity.Theprimaryfiguresexcludethe38 andformistral:7bEN;fortheotherthree,the50–97discordant confidence-zerofallbacks(§6.6).Asasensitivityanalysiswere- peattheRQ1,Δ andΔ pairsdonotsupportaconfidentmagnitude.Wethereforedonot p s ,andlanguage-effectanalyseswiththose claimthatalleffectsarenegligible:theyaremodestbyℎandnon- fallbacksretainedasincorrectpredictionsandreportbothread- negligibleby𝑔where𝑔isestimable.Thelargestgainisqwen2.5:7b | ings; the excluded | readings | are | in statistical_results.json, | | | | | | | | | ------------------ | -------- | --- | ---------------------------- | --- | --- | --- | --- | --- | --- | --- | PT(Δ=+7.9pp,𝑔=+0.253,F10.812vs.0.733). theretainedonesinstatistical_results_retained.json. | | | | | | mistral:7b | PT is | the only | case where | the pipeline | leans to- | | --- | --- | --- | --- | --- | ---------- | ----- | -------- | ---------- | ------------ | --------- | Ablationcontrasts.Δ s (pipelinevs.two-call)usesthesame wardunderperformance(Δ=−2.6pp,𝑝 =0.071),consistentwith | protocolasRQ1(Wilcoxon,Cohen’sℎ,𝑛=623,𝛼′ | | | | =0.0083)totest | | | | | | | | ---------------------------------------- | --- | --- | --- | -------------- | --- | --- | --- | --- | --- | --- | acapacity-floorhypothesis.Thezero-confidencecountdoesnot whetheritisdistinguishablefromzero.Asadirect,non-subtractive supportit(mistral:7bhasatmost2/625suchfallbacksinanycon- checkwealsocomparethebaselineagainstthetwo-callbaseline, dition),soweuseafinercomplianceproxy:theshareofpredicted- wherethetypedcontractisabsentonbothsides,soasignificant NFoutputswithoutavalidnfr_categoryis7.3%(pipeline)and differencereflectsdecompositionanditsaccompanyingprompts 8.1%(two-call)formistral:7bPT,against1.3–1.4%inEnglishand ratherthanthetypedstate(§6.5). 0.3–2.6%intheothercellswithanon-zerorate,i.e.about5–6×its | SBQS26,Nov10–13,2026,Maringá,PR | | | | | | | | | | Zuppardoetal. | | ------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ------------- | Table4:RQ1,pipelinevs.baseline(primarymetric:macroF1).Acc,F1andMCCexcludeconfidence-zeroparse-failurefallbacks (§6.6).𝑝 𝑊: Wilcoxon signed-rank on the same data.𝑝 𝑀: exact McNemar on 623 pairs retaining the fallbacks (Bonferroni 𝛼′ =0.0083).✓:significantinper-requirementcorrectnessintheprimaryanalysis(fallbacksexcluded,𝑝 <𝛼′);itdoesnot 𝑊 markthemacro-F1bootstrapofTable5.‡:significantonlyinthecorrectnesstestthatretainsfallbacks(𝑝 <𝛼′).𝑔:paired 𝑀 Cohen’s𝑔;†:intervalspansseveralmagnitudebands,notreliablyestimable. | | | | | Baseline | Pipeline | | | | | | | --- | ----- | --- | --- | -------- | -------- | --------- | ----------------- | --- | --- | ---- | | | Model | L | Acc | F1 MCC | Acc F1 | MCC ΔF1 ℎ | 𝑝 𝑊 | 𝑝 𝑀 | 𝑔 | Sig. | | | | | | | | | 3.3×10−7 6.4×10−7 | | | ✓ | qwen2.5:7b PT 0.733 0.733 0.527 0.814 0.812 0.638 +0.079 +0.193 +0.253 +0.119† qwen2.5:7b EN 0.784 0.784 0.613 0.808 0.807 0.639 +0.023 +0.060 0.059 0.077 – | | | | | | | | 7.4×10−6 7.7×10−6 | | | ✓ | | --- | --- | --- | --- | --- | --- | --- | ----------------- | --- | --- | --- | llama3.1:8b PT 0.640 0.630 0.446 0.696 0.693 0.511 +0.063 +0.119 +0.287 | | | | | | | | | +0.154† | | ✓‡ | | --- | --- | --- | --- | --- | --- | --- | --- | ------- | --- | --- | llama3.1:8b EN 0.728 0.724 0.557 0.730 0.728 0.556 +0.004 −0.011 0.680 0.0073 −0.117† mistral:7b PT 0.602 0.586 0.380 0.579 0.560 0.345 −0.026 −0.046 0.071 0.092 – ✓ mistral:7b EN 0.606 0.591 0.394 0.638 0.628 0.440 +0.037 +0.066 0.005 0.0066 +0.200 Table5:Paired,class-stratifiedbootstrapofmacro-F1differences(𝐵=10,000).Shown:pointestimateandBonferroni-adjusted interval(𝛼′ =0.05/6,i.e.a99.17%percentileinterval);boldwhenthatintervalexcludeszero.𝑛:requirementsvalid(notafallback) forbothsidesofthatcontrast.PointestimatescandifferslightlyfromTables4and9,whichcomputeeachF1onitsownvalid set.Unadjusted95%intervalsareinmacro_f1_bootstrap.json. | | | | | Pipeline−Baseline | | Δ p(2-call−Base) | Δ s(Pipe−2-call) | | | | | --- | --- | --- | --- | ----------------- | --- | ---------------- | ---------------- | --- | --- | --- | Model L Δ[adj.interval] 𝑛 Δ[adj.interval] 𝑛 Δ[adj.interval] 𝑛 qwen2.5:7b PT +0.079[+0.042,+0.119] 621 +0.076[+0.034,+0.115] 623 +0.005[−0.011,+0.020] 621 qwen2.5:7b EN +0.023[−0.009,+0.057] 623 +0.020[−0.013,+0.054] 623 +0.003[−0.005,+0.013] 623 llama3.1:8b PT +0.063[+0.028,+0.100] 623 +0.053[+0.018,+0.087] 623 +0.011[−0.003,+0.026] 623 llama3.1:8b EN −0.005[−0.041,+0.028] 590 −0.003[−0.036,+0.029] 590 −0.002[−0.013,+0.010] 623 mistral:7b PT −0.027[−0.066,+0.011] 623 −0.024[−0.062,+0.015] 623 −0.003[−0.014,+0.006] 623 mistral:7b EN +0.037[+0.003,+0.073] 622 +0.024[−0.008,+0.058] 621 +0.013[+0.002,+0.027] 621 Englishrate(§6.6).Wepresentthecapacityfloorasahypothesis, llama3.1:8bresultalsodependsontheexclusionpolicy:withthe notatestedmechanism. 33ENfallbacksretained,theMann-Whitneytestgives𝑝 =0.063 AnswertoRQ1:Onrequirementsvalidforbothsides,thepaired andnolongersurvivescorrection,whilethepairedtestissignificant (𝑝 =0.0009). bootstrapsupportsaMacro-F1differenceinfavourofthepipelinein 3of6conditions(Bonferroni-adjusted).Inper-requirementcorrect- | | | | | | | Table6:Language-effectresults.𝑝 | | 𝑀𝑊:Mann-WhitneyU(un- | | | | --- | --- | --- | --- | --- | --- | ------------------------------- | --- | -------------------- | --- | --- | ness,theWilcoxontestagreesinthesame3,andwiththefallbacks paired,originalanalysis;fallbacksexcluded).𝑝 retainedtheMcNemartestfindsadifferencein4of6.Effectsizes 𝑀:exactMcNe- aremodestbyCohen’sℎbutmediumtolargebythepaired𝑔where maron623PT/EN-alignedpairs(fallbacksretained);bold= | | | | | | | significantafterBonferroniover9comparisons(𝛼′ | | | | =0.0056). | | --- | --- | --- | --- | --- | --- | --------------------------------------------- | --- | --- | --- | --------- | estimable.Thedifferencedependsonthemodel’sabilitytoproduce Accuraciesfollowtheexclusionpolicyof§6;fourofthe18 structuredoutput;modelsbelowacompliancethresholdshowno cellsareaffected(§6.6). gainfromthespecialisedtwo-callpipeline. | 6.2 | ComplementaryAnalysis:PTvs.EN | | | | | | | | ℎ 𝑝 | 𝑝 | | --- | ----------------------------- | --- | --- | --- | --- | ----------- | --- | --- | --- | --- | | | | | | | | Arch. Model | PT | EN | 𝑀𝑊 | 𝑀 | Conditions Baseline qwen2.5:7b 0.734 0.785 −0.120 0.034 3.0×10−5 | | | | | | | Baseline llama3.1:8b | 0.640 0.729 | −0.191 | <0.001 | 0.0009 | | ------------------------------------------ | -------------- | ------------ | ------ | ------------------ | ------------------- | -------------------- | ----------- | ------ | ------ | -------- | | Because | PT and | EN classify | the | same requirements, | the paired | | | | | | | | | | | | | Baseline mistral:7b | 0.602 0.607 | −0.010 | 0.862 | 0.801 | | analysis | is the | appropriate | one: | aligning them | by requirement | | | | | | | | | | | | | Pipeline qwen2.5:7b | 0.815 0.809 | +0.015 | 0.793 | 0.913 | | textandapplyingtheexactMcNemartest(column𝑝 | | | | | 𝑀 ofTable6; | | | | | | | | | | | | | Pipeline llama3.1:8b | 0.697 0.730 | −0.075 | 0.188 | 0.011 | | 𝑛 = | 623, fallbacks | retained, | 𝛼′ | = 0.0056) | shows a significant | | | | | | | | | | | | | Pipeline mistral:7b | 0.579 0.638 | −0.121 | 0.034 | 9.1×10−6 | | PT<EN | gap in | five of nine | cells: | the baseline | with qwen2.5:7b | | | | | | | | | | | | | Two-call qwen2.5:7b | 0.811 0.806 | +0.012 | 0.829 | 0.824 | andllama3.1:8b,thepipelinewithmistral:7b,andthetwo-call | | | | | | | Two-call llama3.1:8b | 0.687 0.732 | −0.099 | 0.081 | 0.0004 | | --- | --- | --- | --- | --- | --- | -------------------- | ----------- | ------ | ----- | ------ | baselinewithllama3.1:8bandmistral:7b.Englishisthebetter Two-call mistral:7b 0.583 0.626 −0.090 0.115 0.0004 languageineverysignificantcell,andonlyqwen2.5:7bislanguage- | insensitiveinbothdecomposedarchitectures(𝑝 | | | | | =0.91and0.82). | | | | | | | ------------------------------------------ | --- | --- | --- | --- | -------------- | --------------- | -------------- | --- | ------ | -------------- | | | | | | | | Language-effect | finding: Under | the | paired | analysis, Por- | AnearlierunpairedMann–Whitneyanalysis,keptinTable6for tugueseisassociatedwithsignificantlyloweraccuracythanEnglish comparisononly,reachedsignificanceaftercorrectiononlyforthe in5of9architecture–modelcells,includingdecomposedarchitec- | baselinewithllama3.1:8b(ℎ=−0.191,𝑝 | | | | <0.001).Thatbaseline | | | | | | | | ---------------------------------- | --- | --- | --- | -------------------- | --- | --- | --- | --- | --- | --- | turesformistral:7bandllama3.1:8b.Wethereforedonotclaim BeyondAccuracy:Decompositionvs.Typed-StateQA SBQS26,Nov10–13,2026,Maringá,PR thatdecompositionattenuatesthelanguageeffect,norpresentthe ofallfourpairingsofthetwoexecutions(PT+0.022to+0.041,EN +0.052to+0.097),eachexceedingthe𝜅variationbetweenrepeated resultsasevidenceofmultilingualportability.Thedatasupportonly runs(atmost0.011inPTand0.027inEN).Withtwoexecutions thatqwen2.5:7bislanguage-insensitiveunderbothdecomposed architecturesandthatthebaselinellama3.1:8bgapisnotspecific thenoiseestimateiscoarse,andthePTincrementroughlyhalved tothatarchitecture.Becauserequirementandpromptlanguage (item-bootstrap95%interval[+0.003,+0.041]),soweregarditasa changetogether,andthePortuguesetextisamachinetranslation, reproducibleassociation,notapreciselyestimatedeffect.Wedonot thegapcannotbeattributedtoeither(§7). attributeittothetypedcontract:thevalidationboundaryrejected norealoutput,confidenceroutingaffectedatmostthreeof3,750 6.3 MoSCoWConsistency:ASoftwareQuality prioritisations(§4.5),andthetwoarchitecturesdifferon0.6–2.1%of Observation typesandupto13%ofpriorities,whichpointstoexecution-order | | | | | | | | | and run-to-runeffects | | that | a single-rundesign | | cannot | | separate | | --- | --- | --- | --- | --- | --- | --- | --- | --------------------- | --- | ---- | ------------------ | --- | ------ | --- | -------- | BecausePROMISENFR+providesnoMoSCoWgroundtruth,we fromthecontract.Wedidnottestwhethertheincrementtranslates assessprioritisationasinter-modelbehaviouralconsistency, operationalisedasFleiss’𝜅overthethreemodelswithineach(ar- intoanoperationalbenefit. chitecture×language)stratum(Table7).Itisapotentialproxyfor | | | | | | | | | 6.4 NFRSubcategoryF1(Diagnostic) | | | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | -------------------------------- | --- | --- | --- | --- | --- | --- | --- | theReliabilitycharacteristicof§2,notameasureofprioritisation quality:modelsmayagreeandstillbewrong,andwehaveneither Subcategoryrecalliscomputedoverthe272(of370)NFrequire- prioritygroundtruthnorpractitionerevaluation. mentsalsoclassifiedcorrectlyasNF,andprecisionagainstall625 predictions,soasubcategorylabelemittedonarequirementmis- Table7:Inter-modelMoSCoWconsistency,Fleiss’𝜅(3raters, classifiedasFRcountsagainstthatlabel’sprecision(§5.6).Inthe 6strata),fullfour-labelpriorityset.𝑛:requirementsaligned bestcondition(qwen2.5:7bpipelineEN),amongcategorieswith | | | | | | | | | 𝑛 ≥ 30,SEachievesthehighestF1(0.916;𝑃 | | | | | = 0.927,𝑅 | | = 0.905, | | --- | --- | --- | --- | --- | --- | --- | --- | ------------------------------------- | --- | --- | --- | --- | --------- | --- | -------- | acrossallthreeraters(samefilterasRQ1). | | | | | | | | | 𝑛 | | | (0.701) | (0.566), | | | (0.333) | | --- | ------------ | --- | --- | --- | --- | ------- | --- | --------------- | --- | -------- | ---------- | -------- | -------- | ----- | ------- | | | | | | | | | | = 42), followed | | by PE | and | LF | | while | O | | | | | | | | | | and US (0.429; | 𝑃 | = 1.000, | 𝑅 = 0.273) | are | weakest. | The | US im- | | | Architecture | | L | 𝑛 | 𝜅 | Interp. | | | | | | | | | | balanceindicatesaconservativemodelthatlabelsUsabilityonly Baseline PT 623 −0.1406 poor whenconfident,consistentlyacrossarchitectures,soitreflectsthe | | Baseline | | EN | 623 −0.1220 | | poor | | | | | | | | | | | --- | -------- | --- | --- | ----------- | --- | ---- | --- | ---------------------- | --- | --- | --------- | ------ | ---- | --------- | ------ | | | | | | | | | | model’s representation | | of | the class | rather | than | an effect | of de- | Two-call PT 623 +0.1360 slight composition.ThesepatternsdonotaffecttheRQ1andablation Two-call EN 623 +0.1695 slight conclusions(binarymacroF1)butmarkaboundaryforourqual- Pipeline PT 623 +0.1768 slight ityclaims:validationtiming(§6.6)targetsstructuralconformance, Pipeline EN 623 +0.2388 fair notsemanticcorrectnesssuchascategoryambiguity.Per-category values(allconditions)andtheMust-labelratespermodelarein Thenegative𝜅inbothbaselinestratareflectssystematicstrat- subcategory_errorprop.jsonandkappa_table.json. egydivergence:inthePTbaseline,llama3.1:8bandmistral:7b | | | | | | | | | 6.5 RQ2:Decompositionvs.Typed-StateQuality | | | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | ------------------------------------------ | --- | --- | --- | --- | --- | --- | --- | collapsetoaMustrateof99%and97%whileqwen2.5:7bspreads Assurance evenly(46%),aconfigurationFleiss’𝜅scoresasrandomdisagree- mentbutwhichshowswhichmodelsfailtodiscriminate.TheMust rateofthosetwomodelsfallsto0.90/0.86(two-call)and0.87/0.85 Table8:Whateachcontrastchangesandpermitsustocon- clude. | (pipeline), | while | qwen2.5:7b | | stays at | 0.41–0.42; | the whole | pro- | | | | | | | | | | ----------- | ----- | ---------- | --- | -------- | ---------- | --------- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | gressionisdrivenbythetwoweakermodelsreducingtheirMust- | collapseateachstep. | | | | | | | | Contrast | Whatchanges | | | Whatitsupports | | | | | ----------------------------------------------- | --- | --------- | ---------- | ---------- | --- | ------------ | ----- | ------------- | ------------- | ---- | -------- | -------------- | ---------- | ----- | ---- | | Thesixstratashowamonotoneprogression:baseline(𝜅 | | | | | | ≤−0.12) | | | | | | | | | | | | | | | | | | | Δ p: baseline | calls, | task | prompts, | F1 | associated | with | the | | → two-call | | (≈ +0.15) | → pipeline | (≤ +0.24). | | The two-call | base- | | | | | | | | | | | | | | | | | | →two-call | stagedcontext | | | variant | (not | which | ele- | line,withthepipeline’spromptsbutnotypedstate,alreadyre- ment) | coversmostoftheimprovement;thepipelineaddsΔ𝜅 | | | | | | | +0.04 | | | | | | | | | | -------------------------------------------- | --- | --- | --- | --- | --- | --- | ----- | ------------- | ---------- | --- | --------- | ----- | --- | ------ | ------- | | | | | | | | = | | Δ s: two-call | validation | | boundary, | bound | on | the F1 | differ- | to+0.07.UnlikeΔ forF1,thisincrementisdistinguishablefrom →pipeline confidence,LangGraph ence;QAproperties(not s resamplingnoise:apairedpercentilebootstrap(𝐵 =10,000,Bon- theircause) | ferroni | over | 𝑁 = 2, | 𝛼′ = 0.025; | §5.6) | gives | PT Δ𝜅 = +0.0408, | | | | | | | | | | | ------- | ------------------ | ------ | ----------- | ----------- | ----- | ---------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | [+0.0133,+0.0703], | | | 𝑝 = 0.0004, | | +0.0694, | | | | | | | | | | | 97.5% | CI | | | | and | EN | CI | | | | | | Δ | | | [+0.0310,+0.1083],𝑝 <0.0001.Thisholdsonlywithinthisthree- Table9reportsthethree-waycomparison. p isadifference associatedwiththetwo-callspecialised-promptvariant:itssec- modelpanel(§7).Theprimarydriverofthe𝜅gainisthusdecom- | | | | | | | | | ond call receives | | the already-decided | | requirement_type | | | and | | --- | --- | --- | --- | --- | --- | --- | --- | ----------------- | --- | ------------------- | --- | ---------------- | --- | --- | --- | positionwithitstask-specificprompts,matchingtheF1ablation nfr_category,soitconflatespromptwording,anadditionalcall, (§6.5). andstagedcontext(acall-countcontrolisfuturework).Readwith Theresidualpipeline-vs-two-calldifferenceisreproducible:asec- thatcaveat,itissubstantialin4of6conditions(upto+0.075F1), ondexecutionofthetwelvepipelineandtwo-callconditions(same somostofthepipeline’saccuracyadvantageoverthesingle-call code,seed,andconfiguration;themodelserverwasupdatedin baselineisassociatedwithdecomposition.Underexclusionitis between,§7)reproducedthesignofΔ𝜅(PT+0.022,EN+0.079)and | SBQS26,Nov10–13,2026,Maringá,PR | | | | | | | | Zuppardoetal. | | ------------------------------- | --- | --- | --- | --- | --- | --- | --- | ------------- | negligibleforllama3.1:8bEN(+0.006),butretainingthebase- Δ is+0.006(PT,𝑝 =0.42)and−0.003(EN,𝑝 =0.63),reproducing s line’s33fallbacksraisesitto+0.045;itisnegativeformistral:7b thesmall-modelpattern.Caveats:onemodel,oneexecutionper PT,andlargestforqwen2.5:7bPT,thestrongestinstructionfol- condition;thePTbaselineandpipelineandbothtwo-callrunswere lower.Δ istheresidualF1differencebetweenthetypedpipeline executedonanadjacentdevelopmentlinewhoseagentsdifferin s andthetwo-callbaseline;itbundlestheearliervalidationpoint, categorynormalisation,JSONextraction,andconfidenceforward- execution-ordereffects,andconfidencerouting(triggeredononly ing(noneaffectsthebinaryaccuracyreported,asnoclassification | threeprioritisations,§4.5).Itissmallinallconditions(|Δ | | | |≤0.013), | | | | | | | ------------------------------------------------------ | --- | --- | --------- | --------------------------------------------------------- | --- | --- | --- | --- | | | | s | | fallbacksoccurred),andonlytheENbaselineandpipelinewerere- | | | | | inconsistentindirection,andessentiallyunchangedwhenfallbacks executedwiththeartifact’sagents;and32pipelineand37two-call areretained(largestshift0.003). Englishprioritisations(5–6%)fellbacktoCouldHavebecauseof Onper-requirementcorrectness,apairedWilcoxontest(Bonfer- atrailingcommainthemodel’sJSON,whichdoesnotenterthe roni𝛼′ =0.0083)findsfiveofsixΔ comparisonsnotsignificant; accuracyfiguresbutwouldaffectMoSCoWanalyses. s | thesixth(mistral:7bEN,𝑝 | = 0.0082)narrowlyreachessignifi- | | | | | | | | | ----------------------- | -------------------------------- | --- | --- | --- | --- | --- | --- | --- | cancedespiteasmalleffect(ℎ=+0.023).TheMacro-F1bootstrap 6.6 ValidationTimingandSchemaCompliance ofTable5agrees:onlythatconditionhasanadjustedintervalex- The clearest evidence concerning the typed state’s quality- cludingzero.Thepipelineandthetwo-callbaselinedisagreeon assurancepropertiesisadifferenceinwhenstructuralvalidation only4–15requirementspercondition,sotheintervalof𝑔spans | | | | | occurs, not | a metric. In | all three architectures, | | a parse or | | --- | --- | --- | --- | ----------- | ------------ | ------------------------ | --- | ---------- | severalmagnitudebandsinfiveofsixconditions;onlymistral:7b enum-conversionfailureiscaughtinsidetheresponsibleagent’s EN(7vs.0,𝑔=+0.50,[+0.09,+0.50])isreliablynon-negligible.We | | | | | _parse_response | and replaced | with a | conservative | default | | --- | --- | --- | --- | --------------- | ------------ | ------ | ------------ | ------- | thereforereporttheseeffectsassmallinF1andnotdistinguishable, (FUNCTIONAL, confidence=0.0); no malformed value reaches a atthis𝑛,fromeffectsofthesizetheintervalallows. downstreamagent’scontext.Thisrobustnessisnotattributableto | AsmallΔ doesnotshowthatthetypedcontracthasnoquality | | | | | | | | | | --------------------------------------------------- | --- | --- | --- | ------------------------------------------------------- | --- | --- | --- | --- | | s | | | | thetypedcontract:allfouragentsusethesameregex-basedJSON | | | | | value;itshowsthatF1registersnodifference,whileotherinstru- extractionandfallback,duplicatedacrossthecodebase. mentsdo(§6.3for𝜅 and§6.6forvalidationtiming).Itdoesnot The typed contract’s verifiable contribution is validation establishthatthetypedcontractcausesthosedifferences. timing. In the pipeline, ClassificationOutput’s Pydantic Asadirect,non-subtractivecheck,wecomparethesingle-call constraints(e.g.,confidencein[0,1])areenforcedimmediately baselinewiththetwo-callbaselinehead-to-head,sothatthetyped ClassificationAgent | | | | | after the | | call, before | the result | enters | | ------------------------------------------------ | --- | --- | --------- | --------- | --- | ------------ | ---------- | ------ | | contractisabsentonbothsides(sameprotocolasRQ1,𝛼′ | | | =0.0083). | | | | | | PipelineState.Inthetwo-callablationtheintermediateisaplain Decompositionaloneissignificantin2of6conditionswithfallbacks @dataclass,andPydanticvalidationoccursonlywhenthefinal | excluded(qwen2.5:7bPT,ℎ | =+0.184,𝑔 | =+0.250;llama3.1:8b | | | | | | | | ----------------------- | --------- | ------------------- | --- | --- | --- | --- | --- | --- | outputisassembledafterthesecondcall.Thetypedcontractthus PT,ℎ=+0.099,𝑔=+0.254;bothnegligiblebyℎandlargeby𝑔),and validatesoneLLMcallearlier,anarrowV&Vproperty:aviolation in3of6whenthe33baselinefallbacksareretained(llama3.1:8b couldbeactedonbeforethesecond,moreexpensivecall. | EN,ℎ = +0.092,McNemar𝑝 | = 0.0038),the | sameflipthat | RQ1 | | | | | | | ---------------------- | ------------- | ------------ | --- | --- | --- | --- | --- | --- | OnthePROMISENFR+gridthisdifferenceneversurfacedasa shows.ThisreproducestheRQ1patternwithoutthesubtraction failure:noconfidenceortypevaluefelloutsideitsboundsinanyof | logicofΔ andΔ | | | | | | | | | | ------------- | --- | --- | --- | --- | --- | --- | --- | --- | p s ,strengtheningthecasethatwherethepipeline the18runs.Parse-failurefallbacksarerare:38outputsacross4runs beatsthebaseline,thegainisassociatedwithdecompositionand (33baselinellama3.1:8bEN,5.3%;2pipelineqwen2.5:7bPT;1 itspromptsratherthanwiththetypedcontract. pipelinemistral:7bEN;2two-callmistral:7bEN).Weexclude AnswertoRQ2:MostoftheF1differencebetweenthepipeline themfromTables4,6,and9andreporttheretainedreadingas andthesingle-agentbaselineisassociatedwithdecompositionand | | | | | a sensitivity | analysis (§6). | The zero-confidence | count | is a poor | | ------------------------- | ----------------- | --- | --- | ------------- | -------------- | ------------------- | ----- | --------- | | itstask-specificprompts(Δ | ∈ [+0.020,+0.075] | | | | | | | | p in4of6condi- proxyforstructured-outputcompliance;afineroneistheshareof tions),corroboratedbythedirectbaseline-vs-two-callcomparison; predicted-NFoutputswithnovalidnfr_category:formistral:7b wecannotseparatepromptwordingfromtheextracallandstaged itis7.3%(pipeline)and8.1%(two-call)inPortugueseagainst1.3– context.Thepipeline-minus-two-calldifferenceissmallanddirec- | | | | | 1.4% in English, | and no other | condition | exceeds 2.6%, | which is | | ---------------------- | ----------------------------------- | --- | --- | ---------------- | ------------ | --------- | ------------- | -------- | | tionallyinconsistent(Δ | ∈ [−0.003,+0.013]),andthediscordant | | | | | | | | s whatthecapacity-floorreadingof§6.1relieson.Thethreepipeline pairsaretoofewtobounditsmagnitude.Twonarrowpiecesof fallbacksaretheonlyoutputswithconfidencebelow0.70,soonly3 evidenceconcernthetypedcontract’squality-assuranceproperties: of3,750prioritisationsreceivedtheconservative-labelinstruction; earlierstructuralvalidation(§6.6,13injectedfaults)andastatisti- routingwaseffectivelyinert,andnoreportedresultdependsonit. callyconfirmed,reproducibledifferenceininter-modelMoSCoW | | | | | To verify | the boundary | independently | of the | grid, we | | --- | --- | --- | --- | --------- | ------------ | ------------- | ------ | -------- | consistency(§6.3)whosecausethedesigndoesnotisolate. | | | | | ran a systematic | fault-injection | campaign | through | the | | --- | --- | --- | --- | ---------------- | --------------- | -------- | ------- | --- | Supplementarycheckwithalargermodel.Totestwhether real agents of each architecture with a scripted LLM thepipeline’sgaindependsonthesmallmodels’compliance,we (run_fault_injection_campaign.py). It injects 13 faults ranthethreearchitecturesongpt-4.1-mini(AzureAIFoundry), into the model’s JSON, in either call: six value-range viola- outsidethe18-conditiongrid,withthesameprompts,seed,and (confidence 1.4, −0.5, priority_score 1.5, −0.1; | | | | | tions | | NaN; | | | | --- | --- | --- | --- | ----- | --- | ---- | --- | --- | temperature;noclassificationoutputfellback.Thepipelineoutper- priority_rank0)andsevenparse-levelfaults(invalidormissing | formsthebaselineinaccuracy(PT+0.019,McNemar𝑝 | | =0.043;EN | | | | | | | | -------------------------------------------- | --- | --------- | --- | --- | --- | --- | --- | --- | enumeration,wrongtype,truncatedJSON).Thetypedcontract +0.034,𝑝 <10−4,𝑛=623;onlyENsurvivesBonferroniovertwo addsnothingforparse-levelfaults,whichthetwo-callbaselineand languages,𝛼′ =0.025).Thegainisassociatedwithdecomposition: thepipelinehandleidentically(aflaggedfallbackaftertwocalls). | Δ is+0.013(PT,𝑝 | = 0.17)and+0.037(EN,𝑝 | < 10−4),whereas | | | | | | | | --------------- | --------------------- | --------------- | --- | --- | --- | --- | --- | --- | p BeyondAccuracy:Decompositionvs.Typed-StateQA SBQS26,Nov10–13,2026,Maringá,PR Table9:Decompositionablation(macroF1).Δ 𝑝 =F1(two-call)−F1(baseline),adifferenceassociatedwiththetwo-callvariant (promptwording,extracall,andstagedcontextareconfounded);Δ 𝑠=F1(pipeline)−F1(two-call).Excludeddropstheconfidence- zerofallbacks(§6.6);Retainedkeepsthemaserrors(𝑛=625;ΔcomputedfromunroundedF1).𝑏/𝑐:requirementscorrectonly forthepipeline/onlyforthetwo-callbaseline.𝑔:pairedCohen’s𝑔forΔ 𝑠;†:notreliablyestimable(4–15discordantpairs). | | | | | | | Excluded | | Retained | | | | | --- | --- | ----------- | --- | -------- | ----------- | -------- | ------------- | -------- | ----------- | ------ | --- | | | | | | | | Δ | Δ | Δ | Δ 𝑏/𝑐 | 𝑔 | | | | | Model | | L Base | 2Call Pipe | 𝑝 | 𝑠 | 𝑝 | 𝑠 | | | | | | qwen2.5:7b | | PT 0.733 | 0.808 0.812 | +0.075 | +0.004 +0.075 | +0.001 | 8/7 +0.033† | | | | | | qwen2.5:7b | | EN 0.784 | 0.804 0.807 | +0.020 | +0.003 +0.020 | +0.003 | 3/1 +0.250† | | | | | | llama3.1:8b | | PT 0.630 | 0.682 0.693 | +0.053 | +0.011 +0.053 | +0.011 | 8/2 +0.300† | | | | | | llama3.1:8b | | EN 0.724 | 0.730 0.728 | +0.006 | −0.002 +0.045 | −0.002 | 3/4 −0.071† | | | | | | mistral:7b | | PT 0.586 | 0.563 0.560 | −0.023 | −0.003 −0.023 | −0.003 | 1/3 −0.250† | | | | | | mistral:7b | | EN 0.591 | 0.614 0.628 | +0.023 | +0.013 +0.024 | +0.013 | 7/0 | +0.500 | | Thedifferenceliesinthevalue-rangeviolations(Table10).The confirmeddifferenceininter-modelMoSCoWconsistencywhose typedpipelinerejectseachwherethetypedobjectisbuilt,detecting causewedidnotisolate.Wedonotclaim,becausewedidnottest, classification-stagefaultsafterthefirstcallandprioritisation-stage benefitssuchaspredictablebehaviourundermodelsubstitutionor faults after the second, and keeps the requirement as a flagged auditableintermediatedecisions,andeventhesewoulddepend fallbackaftertwocalls.Theunvalidatedtwo-callbaselinepasses on the model satisfying the structured-output contract. For thesamevaluesthroughitsdataclassandfailswhentheoutput mistral:7bPT(§6.1,§6.6),parsefailuresfallbacktoFUNCTIONAL model is assembled, after the second call; because that failure rather than NF, silently biasing misclassifications toward the propagatesthroughtheretrywrapper,bothcallsarerepeatedup functional class; since the model has no more zero-confidence tothreeattempts,consumingsixcallsbeforetherequirementis outputsthantheothersbut5–6×moreNFpredictionswithout droppedwithoutafailurerecord.Thesingle-callbaselinecontains a valid category in Portuguese, these failures are subtler than all13faultsatitsonecall,andnoarchitectureletanout-of-range malformedresponses.Structured-outputcomplianceisthereforea valuereachtheoutput.Thecampaignissynthetic:itcharacterises prerequisiteforbenefitingfromdecomposition. whereeacharchitecturedetectsafaultandwhatthatcosts,not Practically,teamsinterestedmainlyinF1shouldinvestinbetter howoftensuchfaultsoccur;thatearliervalidationpreventsan promptsratherthanatypedcontract,whereasapplicationsthat operationalfailureremainsanuntestedhypothesis.Thesix-call valueearlierstructuralvalidationorcross-modelconsistencymay costandthelossintheunvalidatedarmpartlyreflecttheshared preferthepipelinedespiteitslimitedaccuracygainoverthetwo- retrypolicy,whichretriesadeterministicvalidationerror,sothe callbaseline.Ourpairedanalysisdoesnotsupportdeployingthe contrastisbetweenacontainedandanuncontainedfailurerather samearchitecturebilinguallywithoutadaptation(§6.2).Although thanbetweendetectionandnon-detection. thepipelinemakestwosequentialcalls,localexecutiontimeswere comparabletothebaseline(qwen2.5:7bPT:3529svs.3415sfor Table10:Fault-injectioncampaign:outcomeandLLMcalls 𝑛 =625);hostedAPIsmayapproacha2×wall-clockcost,which | consumedperfault(13faults:6value-range,7parse-level). | | | | | | wedidnottest. | | | | | | | ----------------------------------------------------- | --- | --- | --- | --- | --- | ------------- | ----------------- | --- | ------------ | ------------- | ------ | | | | | | | | | Internal threats. | | PROMISE NFR+ | has no MoSCoW | ground | Architecture Value-range(6) Parse-level(7) truth,soprioritisationqualityisoperationalisedasinter-model stability,reflectingreliabilityratherthancorrectness;somerequire- | | Baseline | flagged,1call | | flagged,1call | | | | | | | | | --- | -------- | ------------- | --- | ------------- | --- | --- | --- | --- | --- | --- | --- | mentsalsoadmitseveralplausibleNFRcategorieswhilethedataset | | Two-call | dropped,6calls | | flagged,2calls | | | | | | | | | --- | -------- | -------------- | --- | -------------- | --- | --- | --- | --- | --- | --- | --- | givesonelabel.Theprincipalthreatistheprompt–architecture | | Pipeline | flagged,2calls | | flagged,2calls | | | | | | | | | --- | -------- | -------------- | --- | -------------- | --- | --- | --- | --- | --- | --- | --- | confound:thebaselinecombinesbothsubtasksinoneprompt,so Δ p isadifferenceassociatedwiththetwo-callvariant,bundling Twoconclusionsfollow.Structured-JSONcomplianceisachieved promptwording,anextracall,andstagedcontext,andacall-count- independentlyineacharchitecture,notthroughonecentralised controlled arm would be needed to separate them. Δ likewise s component.Andvalidationtiming,notfailurecontainment,iswhat bundlesthevalidationboundary,confidencehandling(whichfired distinguishesthetypedpipelinefromtheunvalidatedablation;the for3of3,750predictions),andLangGraphorchestration,soitis isolatedeffectofconfidenceroutingcouldnotbeevaluated,since apipeline-versus-two-callcontrastratherthananisolatedtyped- itfiredononlythreefallbacks. contracteffect,anditsrangeisanupperboundontheF1difference associatedwiththetypedcontract.Cohen’sℎplacesallRQ1effects 7 DiscussionandThreatstoValidity below0.20butignoresthepairing;thepaired𝑔ismediumorlarge The experiments indicate that the typed contract is accom- whereestimable,sowedonotcalltheeffectsnegligible.ForΔ s panied by little difference in classification performance the4–15discordantpairsaretoofewforanyreliablepairedeffect (Δ [−0.003,+0.013]). s ∈ The evidence for what it is accom- size.Excluding38fallbacksisananalyst’sdecision,henceboth paniedbyislimitedtotwoproperties:earlierstructuralvalidation, readings;itchangesoneRQ1verdict(llama3.1:8bEN)andone | verified | by a 13-fault | injection | campaign, | | and a statistically | | | | | | | | -------- | ------------- | --------- | --------- | --- | ------------------- | --- | --- | --- | --- | --- | --- | SBQS26,Nov10–13,2026,Maringá,PR Zuppardoetal. Δ estimate.Decodingattemperature0isnotbit-deterministic: ACKNOWLEDGMENTS p primaryresultscomefromoneexecutionpercondition,andthe Forpartiallysupportingthiswork,wewouldliketothankINES.IA onerepeatedexecution(pipelineandtwo-call)ranonanewerOl- (NationalInstituteofScienceandTechnologyforSoftwareEngi- lamarelease(0.35.1vs.0.24.0),soitsvariationmixesrun-to-run neeringBasedonandforArtificialIntelligence),https://www.ines. andserver-versioneffects;differencesbelowaboutonepointof org.br,CNPqgrant408817/2024-0. F1(and0.03in𝜅)shouldnotbeinterpreted.Thefault-injection evidencerestson13syntheticfaults(sixofthemvalue-rangeviola- REFERENCES tions,whichextendtheearlierfive-valueprobe):itshowswhere [1] PedroCabral,JoãoArruda,CleidsonSouza,andVictorPinto.2025. Evaluat- validationoccursandwhatitcosts,nothowoftensuchfaultsarise. ingLLM-GeneratedUnitTestswithMutationTesting:ChatGPTvsDeepSeek. Per-calltraceswereretainedonlyforthetwo-callsupplementruns, InAnaisdoXXIVSimpósioBrasileirodeQualidadedeSoftware(SãoJosédos Campos/SP).SBC,PortoAlegre,RS,Brasil,120–130.doi:10.5753/sbqs.2025.13792 whichlimitsauditofindividualfailures. [2] LuisF.G.Campos,LeonardoC.Marques,andWalterT.Nakamura.2025.WillAI External threats. All experiments use PROMISE NFR+ and alsoreplaceinspectors?InvestigatingthepotentialofgenerativeAIsinusability inspection.InAnaisdoXXIVSimpósioBrasileirodeQualidadedeSoftware(SBQS threedecoder-onlySLMs(7–8Bparameters)servedthroughver- 2025).SBC. https://sol.sbc.org.br/index.php/sbqs/article/view/38990 sionedOllamatags,whosebehaviourmaychange;compliancerates [3] DaiCleggandRichardBarker.1994.CaseMethodFast-Track:ARADApproach. andthecontributionoftypedcontractsmaydifferforotherdatasets, Addison-Wesley. domains,ormodels,andthe𝜅gainreflectsonlythisthree-model [4] JacobCohen.1988.StatisticalPowerAnalysisfortheBehavioralSciences(2nded.). LawrenceErlbaumAssociates. panel.Thesinglelargermodel(§6.5)isonemodelwithoneexecu- [5] JosephL.Fleiss.1971.MeasuringNominalScaleAgreementAmongManyRaters. tionpercondition.Thelanguagecomparisonvariesrequirement PsychologicalBulletin76,5(1971),378–382. [6] ArshiaHemmat,MohammadrezaSharbaf,ShekoufehKolahdouz-Rahimi,Kevin andpromptlanguagejointly,andthePortuguesetextisamachine Lano,andSobhanY.Tehrani.2025.ResearchDirectionsforUsingLLMinSoftware translation(back-translationBERTScoreF10.9646;624of625pairs RequirementEngineering:ASystematicReview.FrontiersinComputerScience ≥0.90),sotranslationartefactsmaycontributetothePT–ENgap. (2025).doi:10.3389/fcomp.2025.1519437 [7] LeiHuang,WeijiangYu,WeitaoMa,WeihongZhong,ZhangyinFeng,Haotian OnlythepipelineusedLangGraph,althoughthetyped-statepattern Wang,QianglongChen,WeihuaPeng,XiaochengFeng,BingQin,andTing isnotspecifictoit.Bonferronicorrectionwasappliedthroughout, Liu.2025. ASurveyonHallucinationinLargeLanguageModels:Principles, Taxonomy,Challenges,andOpenQuestions.ACMTransactionsonInformation andcategorieswithfewerthan30instanceswereexcludedfrom Systems43,2(2025),1–55.Preprint:arXiv:2311.05232(2023).doi:10.1145/3703155 theprimaryanalyses.Ourconclusionsarethereforeintentionally [8] J.RichardLandisandGaryG.Koch.1977.TheMeasurementofObserverAgree- narrow:fortheevaluateddataset,task,andmodelpanel,themea- mentforCategoricalData.Biometrics33,1(1977),159–174. [9] LangChain.2024.LangGraph:BuildingStateful,Multi-ActorApplicationswith surabledifferencesthataccompanythetypedcontractareearlier LLMs.https://www.langchain.com/langgraph.Softwaredocumentation,accessed structuralvalidationandasmalldifferenceininter-modelconsis- 2026. [10] JunweiLiu,KaixinWang,YixuanChen,XinPeng,ZhenpengChen,Lingming tency,ratherthanimprovementsinclassificationaccuracy. Zhang,andYilingLou.2026.LargeLanguageModel-BasedAgentsforSoftware Engineering:ASurvey.(2026). doi:10.1145/3796507 8 ConclusionandFutureWork [11] MuriloMartin,DanielCoutinho,AndersonUchôa,andJulianaAlvesPereira. 2025. EvaluatingthePotentialofLargeLanguageModelsinSecurity-Related Intheevaluatedsetting,theaccuracydifferenceisassociated SoftwareRequirementsClassification.InAnaisdoXXXIXSimpósioBrasileirode withdecompositionintospecialisedprompts,notwiththe EngenhariadeSoftware(SBES2025).SBC. https://sol.sbc.org.br/index.php/sbes/ article/view/37009 typedstate(§6.1–§6.5).Thetypedstateisaccompaniedbytwo [12] QuinnMcNemar.1947.Noteonthesamplingerrorofthedifferencebetween differencesbeyondaccuracy:containmentofout-of-rangevaluesat correlatedproportionsorpercentages.Psychometrika12,2(1947),153–157. theagentboundary,withclassificationfaultsdetectedonecallear- [13] GokulRejithkumarandPreethuRoseAnish.2025. NICE:Non-Functional RequirementsIdentification,Classification,andExplanationUsingSmallLan- lierina13-faultcampaign(§6.6),andasmall,reproducibleincrease guageModels.InProceedingsoftheIEEE/ACMInternationalConferenceonSoft- ininter-modelMoSCoWconsistency(pairedbootstrap,𝑝 <0.001 wareEngineering:SoftwareEngineeringinPractice(ICSE-SEIP2025).IEEE/ACM. doi:10.1109/ICSE-SEIP66354.2025.00031 inbothlanguages,§6.3).Thedesigndoesnotisolatetheircauses, [14] ReineSantos,IgorSteinmacher,TayanaConte,AnaCarolinaOran,andBruno andthelimitsaredetailedin§7. Gadelha.2025.AdoptionofLLMsinRequirementsEngineering:WhatPractition- FutureWork.(i)Calibratingtheconfidencethreshold,toisolate ersAreWorriedAbout?.InAnaisdoXXIVSimpósioBrasileirodeQualidadedeSoft- ware(SBQS2025).SBC. https://sol.sbc.org.br/index.php/sbqs/article/view/39013 routingfromvalidationtiming.(ii)Acall-count-controlledarmto [15] BrunoSilva,RodrigoNascimento,LuisRivero,GeraldoBrazJúnior,Rodrigo separatepromptwordingfromtheextracallandstagedcontext.(iii) PereiradosSantos,LuizEduardoGalvãoMartins,andDaviViana.2024.Promise+: Largermodelsandalargerraterpanel,totestwhetherΔ grows ADatasetofLabeledSoftwareRequirements.doi:10.5281/zenodo.12805484 s [16] BrunoSilva,RodrigoNascimento,LuisRivero,GeraldoBrazJúnior,Rodrigo withcapacityandtheconsistencygaingeneralises. PereiradosSantos,LuizEduardoGalvãoMartins,andDaviViana.2024.Promise+: ExpandindoaBasedeDadosdeRequisitosdeSoftwarePromise_exp.InAnaisdo ARTIFACTAVAILABILITY XXXVIIISimpósioBrasileirodeEngenhariadeSoftware(SBES2024).SBC. https: //sol.sbc.org.br/index.php/sbes/article/view/30370 The implementation, prompts, run manifests, raw predictions, [17] FrankWilcoxon.1945.IndividualComparisonsbyRankingMethods.Biometrics Bulletin1,6(1945),80–83. analysis scripts, and unit tests are archived on Zenodo (https: [18] MohammadAminZadenoori,JacekDąbrowski,WaadAlhoshan,LipingZhao, //doi.org/10.5281/zenodo.23380444).ThePROMISENFR+fileisnot andAlessioFerrari.2025. LargeLanguageModels(LLMs)forRequirements redistributed;itsMD5hashisrecordedineverymanifest. Engineering(RE):ASystematicLiteratureReview. arXiv:2509.11446. https: //arxiv.org/abs/2509.11446 [19] MohammadAminZadenoori,VincenzoDeMartino,JacekDąbrowski,Xavier Franch,andAlessioFerrari.2025.DoesModelSizeMatter?AComparisonofSmall andLargeLanguageModelsforRequirementsClassification.arXiv:2510.21443. https://arxiv.org/abs/2510.21443