← Back to summary
When LLMs Forget What They Believe: Characterizing State Integrity Collapse in Long-Horizon Dialogues
Converted from PDF with markitdown. Layout may be rough.
When LLMs Forget What They Believe: Characterizing State Integrity
| | | Collapse | | in Long-Horizon | | Dialogues | | | | | |
| --- | --- | -------- | --- | --------------- | ---------- | --------- | --- | --- | --- | --- | --- |
| | | | | YanchengZhu | YifanYuan* | | | | | | |
SchoolofArtificialIntelligence,ShenzhenUniversity,Shenzhen,China
2410673009@mails.szu.edu.cn,yifanyuan@szu.edu.cn
| | Abstract | | | | We | call this | failure | State | Integrity | | Collapse |
| --- | -------- | --- | --- | --- | ------ | ----------- | ------- | -------- | ----------- | --- | -------- |
| | | | | | (SIC). | SIC differs | from | ordinary | forgetting. | | For- |
Largelanguagemodelscanrepeataninstruc-
6202 tcO 22 ]LC.sc[ 1v41313.0162:viXra getting means the model cannot recover a prior
| tionwhilefailingtokeepobeyingit. | | | | Westudy | | | | | | | |
| -------------------------------- | --- | --- | --- | ------- | --- | --- | --- | --- | --- | --- | --- |
this recall–compliance dissociation in long- constraint;SICmeansthemodelrecoversitunder
horizondialogueanddefineStateIntegrityCol- probebutfailstomakeitgovernthenextanswer.
lapse: a model preserves a stated constraint The distinction matters for mitigation. Retrieval,
inmemorybutstopsapplyingittolaterturns.
memorysummarization,orlongercontextwindows
We introduce STATETRACK-BENCH, a con- may help when state is inaccessible, but they do
trolledsuiteofconversationscoveringnumeri-
notdirectlysolvefailuresofstateapplication.
cal,format,role/persona,factual-memory,and
| | | | | | The | practical | stakes | are | high. Consider | | a med- |
| ------------------------ | --- | -------------------- | --- | --- | --- | --------- | ------ | --- | -------------- | --- | ------ |
| logical-ruleconstraints. | | Acrosssevenprovider- | | | | | | | | | |
pinned model snapshots and five dialogue- ical assistant instructed to avoid recommending
medicationscontraindicatedforapatient’scondi-
depthprobes,strictjointretentionrangesfrom
62.4% to 100.0%, with degradation concen- tion,oralegalassistantboundbyaconfidentiality
tratedinsymbolicconstraintsandinoneespe-
| | | | | | constraint. | Ifthemodelcanrestatetherulewhen | | | | | |
| --- | --- | --- | --- | --- | ----------- | ------------------------------- | --- | --- | --- | --- | --- |
ciallybrittlemodel.Aforkedrecallauditshows askedbutviolatesitinthenextresponse,thefailure
that88.6%ofobservedfailuresarebehavioral-
| | | | | | is invisible | to | standard | evaluation | | yet potentially | |
| -------------- | ----------- | ------------- | --------- | ----------- | -------------------- | --- | -------- | ---------------------- | --- | --------------- | --- |
| drift failures | rather | than ordinary | | forgetting; | | | | | | | |
| | | | | | harmfulindeployment. | | | SICcapturesexactlythis | | | |
| excluding | the brittle | outlier | preserves | this di- | | | | | | | |
agnosis(84.4%offailures). Extendedevalua- classoffailure: themodel’sdeclarativeknowledge
anditsgenerativebehaviorbecomedecoupledas
tionsatdepthsupto50turnsconfirmthatthe
| dissociation | persists | beyond | the | core bench- | dialoguestategrows. | | | | | | |
| ------------ | ---------- | ------ | --------- | ----------- | ------------------- | --- | --- | --- | --- | --- | --- |
| mark, while | Structured | State | Anchoring | re- | | | | | | | |
Wecontributeacompactbenchmarkandanaly-
covers up to 24.4 points on immutable con- sisprotocolformeasuringthisgap. STATETRACK-
| straints. | Theresultsshowthatreliabilitycan- | | | | | | | | | | |
| --------- | --------------------------------- | --- | --- | --- | ----- | ------- | ------------- | --- | ---- | --------- | --- |
| | | | | | BENCH | creates | conversations | | that | introduce | one |
notbeinferredfromsingle-turninstructionfol-
| | | | | | persistent | constraint | | and | later require | the | model |
| ------ | ----------------- | ---- | ----- | ------- | ---------- | ---------- | ----- | ---------- | ------------- | --- | ------ |
| lowing | alone; evaluation | must | track | whether | | | | | | | |
| | | | | | to obey | it | under | increasing | dialogue | | depth. |
rememberedconstraintsremainoperationalas
| | | | | | Constraints | | cover | symbolic | and | semantic | cate- |
| -------- | ------------ | ---- | --- | ------------- | ----------- | --- | ----- | -------- | --- | -------- | ----- |
| dialogue | state grows. | Code | and | artifacts are | | | | | | | |
availableat: https://github.com/xzzvsxd/ gories: numerical constraints, format constraints,
| StateTrack. | | | | | role/personaconstraints,factual-memorybindings, | | | | | | |
| ----------- | --- | --- | --- | --- | ----------------------------------------------- | --- | ------------------------- | --- | --- | --- | --- |
| | | | | | andlogicalrules. | | Weevaluatesevenmodelsnap- | | | | |
1 Introduction
shotsatfivedialogue-depthprobesusingdetermin-
Instruction-following evaluation usually asks isticdecodingandfixedjudgeprompts. Thecore
whetheramodelcansatisfyaconstraintinonein- benchmarkcontains125scenario-depthprobesper
teractionorwithinashorttask-orientedexchange. model (25 per category), yielding 875 primary
Thissettingistooweakfordeployedassistantsthat scoredoutputsacrossallmodels. Extendedpilots
| must preserve | goals, | preferences, | | safety bound- | | | | | | | |
| ------------- | ------ | ------------ | --- | ------------- | --- | --- | --- | --- | --- | --- | --- |
atdepthsupto50turnsandintwoadditionallan-
aries, and formatting requirements across many guages(Chinese,Japanese)testwhetherthesame
turns. Amodelthatcanrestateaconstraintbutno diagnosisappearsoutsidethecoresetting.
longerappliesitpresentsadangerousfailuremode: Our findings are threefold. First, strict joint
theuserseesevidenceofmemorybutreceivesbe- retention varies substantially by model: overall
haviorthatviolatestherememberedstate. scoresrangefrom62.4%to100.0%,andthemost
*Correspondingauthor. dramaticdecayisdrivenbyGemini-3.1-Pro. Sec-
ond, symbolic constraints are more brittle than Long-contextdegradation. Long-contextwork
semantic memory: logical-rule and format con- has shown that information in the middle of a
straintsdegradefasterthanfactual-memorybind- promptishardertousethaninformationnearthe
ings. Third,turndepthexposesbehaviorthataggre- beginning or end (Liu et al., 2024). We extend
gateinstruction-followingscoreshide: somemod- thisperspectivefromretrievaltocompliance: the
els remain stable, whereas others lose 10 points relevant information may still be verbally recov-
from early to late turns. Critically, 88.6% of ob- erable, yetfailtocontrolgeneration. Thismakes
servedfailuresareintegrityfailures—caseswhere SICastate-applicationproblemratherthanmerely
themodelcanrecalltheconstraintbutfailstoen- acontext-accessproblem. RULERextendsneedle-
force it—rather than memory failures where the in-a-haystackevaluationwithtracingandaggrega-
constraint is genuinely lost; this remains 84.4% tiontasks,whileL-Evalcombineslong-document
when Gemini-3.1-Pro is excluded. Extended- question answering with open-ended generation
depthandcross-lingualpilotsarethereforetreated evaluation (Hsieh et al., 2024; An et al., 2024).
assupportingevidence,notasaclaimthatallcur- These benchmarks test whether models can re-
rentLLMscollapseuniformly. trieve and use information over long contexts, a
necessarybutnotsufficientconditionforpersistent
Thesefindingscomplementworkoninstruction
| | | | | | | | constraintadherence. | | Recentworkfurthershows | | | | |
| --- | --- | --- | --- | --- | --- | --- | -------------------- | --- | ---------------------- | --- | --- | --- | --- |
following,dialoguebenchmarks,long-contextposi-
tioneffects,andpersistent-instructioncompliance that reasoning performance degrades with input
| | | | | | | | length | even when | all | relevant | information | | is ac- |
| ------------------ | -------- | ------ | ------- | ------------- | --- | ------- | -------- | --------- | ------------- | -------- | ----------- | ------ | ------ |
| in safety | settings | (Jiang | et al., | 2024; | Qin | et al., | | | | | | | |
| | | | | | | | cessible | (Levy | et al., 2024; | | Li et al., | 2024), | and |
| 2024; Budzianowski | | et | al., | 2018; Rastogi | | et al., | | | | | | | |
needle-in-a-haystackprobes(Kamradt,2023)con-
| 2020;Liuetal.,2024). | | | Thebenchmark,evaluation | | | | | | | | | | |
| -------------------- | --- | --- | ----------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
code,andallmodellogsarereleasedtofacilitate firmposition-dependentretrievalfailures. Amodel
| | | | | | | | that correctly | recalls | a | constraint | may | still | fail to |
| --- | --- | --- | --- | --- | --- | --- | -------------- | ------- | --- | ---------- | --- | ----- | ------- |
replication.
enforceitingeneratedoutput,preciselythedisso-
ciationthatmotivatesourrecall-versus-compliance
| 2 RelatedWork | | | | | | | decomposition. | | | | | | |
| ------------- | --------- | --- | ------------ | --- | ----------- | --- | ------------------------------------- | ------ | ---------- | --- | ----- | --- | ------ |
| | | | | | | | Agentmemoryandpersistentinstructions. | | | | | | Re- |
| Instruction | following | | and dialogue | | evaluation. | | | | | | | | |
| | | | | | | | cent agent | memory | benchmarks | | focus | on | recov- |
Instruction-followingbenchmarkshavegrownbe-
| | | | | | | | erability | or tool-mediated | | use; | our | work | isolates |
| --- | --- | --- | --- | --- | --- | --- | --------- | ---------------- | --- | ---- | --- | ---- | -------- |
yondsimplepromptadherencetomulti-constraint
| | | | | | | | whether | remembered | | constraints | remain | | behav- |
| --------------- | --- | -------- | ----- | --- | ---- | ----- | ------- | ---------- | --- | ----------- | ------ | --- | ------ |
| and adversarial | | settings | (Zhou | et | al., | 2023; | | | | | | | |
iorallybinding,whichisacomplementarydiagnos-
| Jiang et | al., | 2024; Qin | et | al., 2024). | | Task- | | | | | | | |
| -------- | -------- | --------- | --- | ----------- | -------- | ----- | --------------------------------- | --- | --- | --- | ------------ | --- | --- |
| | | | | | | | ticlensonlong-horizonreliability. | | | | Automatedred | | |
| oriented | dialogue | corpora | | such as | MultiWOZ | | | | | | | | |
teaminghasexposedharmfulmodelbehaviors,in-
| and SGD | evaluate | slot | tracking | and | policy | deci- | | | | | | | |
| ------- | -------- | ---- | -------- | --- | ------ | ----- | --- | --- | --- | --- | --- | --- | --- |
cludingharmsthatemergeoverthecourseofacon-
| sions (Budzianowski | | et | al., | 2018; Rastogi | | et al., | | | | | | | |
| ------------------- | --- | --- | ---- | ------------- | --- | ------- | --------------------------- | --- | --- | --- | ---------------- | --- | --- |
| | | | | | | | versation(Perezetal.,2022). | | | | Ourbenchmarkdoes | | |
2020),buttheirstateschemaisusuallyexplicitand
| | | | | | | | not use | adversarial | jailbreak | | attempts; | instead, | it |
| -------------- | --- | --------------------------------- | --- | --- | --- | --- | ------- | ----------- | --------- | --- | --------- | -------- | --- |
| short-horizon. | | Oursettinginsteadtargetsfree-form | | | | | | | | | | | |
testswhetherpragmaticaccommodationoverrides
| assistant | behavior | where | a natural-language | | | con- | | | | | | | |
| --------- | -------- | ----- | ------------------ | --- | --- | ---- | --- | --- | --- | --- | --- | --- | --- |
constraintadherenceundernaturalisticsocialpres-
| straint must | remain | active | across | turns. | The | key | | | | | | | |
| ------------ | ------ | ------ | ------ | ------ | --- | --- | ---------------------------------------------- | --- | --- | --- | --- | --- | --- |
| | | | | | | | sure. ThisdistinctionpositionsStateTrack-Bench | | | | | | |
structuraldifferenceisthatslot-fillingoperatesover
| | | | | | | | as a diagnostic | | for deployment | | reliability | | rather |
| --- | --- | --- | --- | --- | --- | --- | --------------- | --- | -------------- | --- | ----------- | --- | ------ |
aclosed,task-specificontology(e.g.,hotelname,
thanared-teamingtool.
| check-in | date), | whereas | our | constraints | are | open- | | | | | | | |
| -------- | ------ | ------- | --- | ----------- | --- | ----- | --- | --- | --- | --- | --- | --- | --- |
domainandlinguisticallyheterogeneous,spanning
| | | | | | | | 3 BenchmarkandMetrics | | | | | | |
| -------------- | --- | -------- | ----------- | --- | ------------ | --- | --------------------- | --- | --- | --- | --- | --- | --- |
| format, style, | | content, | and persona | | requirements | | | | | | | | |
that do not reduce to discrete slot values. More- Eachbenchmarkinstanceisadialoguewiththree
over, DST benchmarks treat state tracking as a phases. In the setup phase, the user states a per-
standalonesub-taskevaluatedwithgolddialogue sistentconstraint. Inthedistractorphase,thecon-
history,whereasourdesignrequiresthemodelto versation proceeds with semantically related but
bothmaintainconstraintrecallandsimultaneously non-identicalrequests. Inthetestphase,themodel
enforce compliance in generated output—a joint mustanswerwhilestillsatisfyingtheoriginalcon-
generativedemandthatDSTpipelinesseparateby straint. Weusefiveconstraintcategories: numer-
construction. ical constraints, format constraints, role/persona
Benchmark scale and coverage. The core
benchmarkcomprises125scenario-depthprobes
permodel(25percategory)across7models,yield-
ing875primaryscoredoutputs. Thisdesignpriori-
tizesdiagnosticprecisionoverrawvolume: each
probeisacarefullyconstructedmulti-turndialogue
prefix with controlled distractor properties, not a
template-generated surface variant. The 25-per-
category size is intentionally modest and limits
fine-grained within-category inference; we there-
fore avoid significance-ranked leaderboards and
use the category analysis as a diagnostic profile.
Figure1: OverviewoftheSTATETRACK-BENCHevalu-
Robustnessisassessedthroughtargetedauditsthat
ationprotocol. Eachdialoguepassesthroughconstraint
injection,distractor-phaseexpansion,andaforkedeval- isolate the main methodological concerns: split
uationbranchthatindependentlymeasuresrecallfidelity task-versus-constraintscoring,signedSICcompo-
andbehavioralcomplianceacrossfiveconstraintcate- nents, a Gemini-excluded depth study, dynamic
goriesandfiveturndepths. update/revocation, a two-constraint scope check,
andprompt-stylesensitivityanalysis(AppendixI;
constraints,factual-memorybindings,andlogical AppendixL).
rules.
Constraint taxonomy. The five categories in-
Operationally,apersistentconstraintmust(i)re-
tentionally mix surface-form and semantic state.
mainactiveacrossturns,(ii)beseparablefromthe
Numerical constraints require a bound, count, or
immediatetask,(iii)admitanoracle-satisfiablere-
threshold to remain exact; format constraints re-
sponseateverytestedturn,and(iv)remainbinding
quiretheoutputtemplatetosurvivedistractorturns;
only until the user updates or revokes it. Thus,
role/persona constraints measure style and voice
“Find a Spanish restaurant” is an immediate re-
persistence;factual-memorybindingstestwhether
quest,whereas“Fortherestofthissearch,recom-
anamedfactoruserpreferenceremainsavailable;
mendonlySpanishrestaurants”establishesstand-
and logical rules require a conditional policy to
ingstate;“Ifnoneareavailable,switchtoFrench
be applied only when its antecedent holds. This
restaurants”updatesthatstate. Thisboundaryex-
separation matters because a model can be good
cludesone-shotrequestsandtreatspersistenceas
at remembering semantic facts while still failing
revisableratherthanirreversible.
atexactstructuralcommitments. Thetaxonomyis
For a dialogue i at turn depth t, let c be the
i
therefore designed to expose which kind of state
activeconstraintandy themodeloutput. Ajudge
i,t
breaks,notonlywhetheralateransweriswrong.
assigns J(y ,c ) ∈ {0,1}, where 1 means the
i,t i
output satisfies both the immediate request and Distractor construction. Distractor turns are
the persistent constraint. The headline metric is
written to be plausible continuations of the con-
thereforestrictjointretention,notpureconstraint
versationwithoutrestatingtheoriginalconstraint
adherence:
verbatim. Ifalaterusermessagerepeatstherule,
1 (cid:88)(cid:88) thetaskdegeneratesintoshort-contextinstruction
R = J(y ,c), (1)
m m,c,t
|C||T| following; if it is unrelated, the conversation no
c∈C t∈T
longerresemblesdeployedassistantuse. Wethere-
for model m, categories C, and turn depths T =
fore use distractors that are topically compatible
{3,5,7,10,12}. We also report early-to-late de-
withthesetupbutdonotcontaindirectlexicalre-
cay, ∆ = R (t=3) − R (t=12). Because J
m m m mindersoftheactiverule. Theturn-depthschedule
is a joint success indicator, R should be read
m samplesearly,middle,andlatepositionsinthedia-
as a conservative deployment metric: a response
logue,lettingusdistinguishimmediateuptakefrom
is counted as retained only if the local task and
sustainedstatemaintenance.
thepersistentstatearebothsatisfied. TheSICde-
composition below is computed from the forked Scoring policy. The judge is deliberately con-
recall/complianceauditandisthepaper’sdiagnos- servative: an output receives credit only when it
ticmeasureofrecall–usedissociation. satisfiesboththecurrentuserrequestandtheper-
sistentconstraint. Aresponsethatremembersthe headlineretentionmetricisdeliberatelyconserva-
rulebutfailstheimmediatetaskisnotcountedas tive: constraint-onlycompliancecanbehigherthan
successful,andaresponsethatanswerstheimmedi- strict joint retention when a model preserves the
atetaskwhileviolatingthepersistentruleisaSIC persistent rule but gives an incomplete answer to
candidate. For numerical and format constraints, theimmediaterequest. Wethereforeusestrictjoint
the decision can often be made by deterministic retentionasthedeployment-facingreliabilityscore
checks; for semantic categories, the fixed judge and SIC as the diagnostic recall–compliance de-
prompt evaluates the response against the full di- composition.
alogue state. We therefore refer to R as strict
m
Inter-annotator agreement. For the
jointretentionthroughouttheresults,andreserve
role/personaandlogical-rulecategorieswhereau-
“constraintadherence”forthepersistent-statecom-
tomatedscoringislessstraightforward,twotrained
ponent used in the forked audit. This distinction
annotators independently labeled a stratified
preventsunrelatedimmediate-taskerrorsfrombe-
sample of 80 outputs (40 per category, balanced
inginterpretedasmemoryloss.
across models and turn depths). Cohen’s κ was
Formal SIC metric. SIC is computed per in- 0.87 for role/persona and 0.82 for logical-rule
stance,notasadifferenceofaggregatemeans. For judgments. Disagreementswereadjudicatedbya
a shared dialogue prefix h , we create two non- third annotator. Agreement between the human
i,t
interactingbranches: thecompliancebranchasks consensusandtheautomatedjudgewasκ = 0.91
the next task question and yields b ∈ {0,1}; for role/persona and κ = 0.84 for logical-rule
m,i,t
therecallbranchasksthemodeltorestatetheac- items. The automated judge tends to be slightly
tiveconstraintandyieldsr ∈ {0,1}. Thetwo stricter than human raters on edge cases where
m,i,t
branchessharethesamehistorybuttherecallprobe partial persona maintenance is present, which is
isneverinsertedintothescoredcompliancetrajec- consistentwithourconservativedesignintent.
tory. ForasetP ofprobedmodel–dialogue–turn
Robustnesstodecodingregime. Toverifythat
triples,
SICisnotanartifactofdeterministicdecoding,we
re-evaluatedthreemodelsattemperature0.7with
1 (cid:88)
SIC m = |r m,i,t −b m,i,t |, (2) 3 samples per item. The maximum shift was 2.1
|P|
(i,t)∈P points;modelorderingandcategory-levelpatterns
1 (cid:88) werefullypreserved(detailsinAppendixE).
SIC+ = 1[r = 1∧b = 0],
m |P| m,i,t m,i,t
(i,t)∈P 4 ExperimentalSetup
(3)
We evaluate seven model snapshots using deter-
1 (cid:88)
SIC− m = |P| 1[r m,i,t = 0∧b m,i,t = 1]. ministic decoding: GPT-5.4, GPT-5.4-mini,
(i,t)∈P Claude-Sonnet-4-6, Gemini-3.1-Pro,
(4) DeepSeek-v4-Pro, DeepSeek-v4-Flash, and
Qwen3.6-Plus. Model identifiers are treated as
SIC+ is behavioral drift: the model can articu- pinnedprovidersnapshotsinourlogsratherthan
late the constraint but fails to follow it. SIC− is as claims about final public product names; the
declarativefade: themodelfollowstheconstraint logged endpoint identifiers are listed in Table 1.
behaviorallybutcannotstateit. Becausethemetric All prompts, judge prompts, and random seeds
isaveragedafterper-instancecomparison,opposite arefixed. Weusepairedbootstraptestsformodel
dissociationscannotcancelout. comparisons following common NLP evaluation
practice (Koehn, 2004). Fixed configurations
Strict-joint versus constraint-only audit. Be-
and seeds also reduce the experimental variation
cause strict joint retention counts both local task
highlightedbyDodgeetal.(Dodgeetal.,2020).
success and persistent-state compliance, we ad-
ditionally ran a compact split audit that scores Dialoguedesign. Eachbenchmarkinstancefol-
thesefactorsseparately(AppendixI).Thisauditre- lows the same three-stage structure: a setup turn
portsimmediate-tasksuccess,constraint-onlysuc- introduces the persistent constraint, several dis-
cess, strict joint success, and the forked recall la- tractorturnsexpandthedialoguecontextwithout
bels on the same 70 outputs. It confirms that the changingtheconstraint,andalatertestturnchecks
Table1: Loggedmodelsnapshotidentifiersusedforthe Table 2: Average strict joint retention by turn depth
primarybenchmarkruns. Closed-providernamesmay acrossallconstraintcategories.
driftinfuturereleases;thesestringsarethereproducibil-
ityhandlesstoredwithourmodellogs.
| | | | | | | Turn | | | 3 | 5 | 7 10 | 12 |
| ------------- | --- | ------------------------- | --- | --- | --- | -------------------- | --- | --- | ---- | ---- | --------- | ---- |
| | | | | | | Strictjointretention | | | 90.9 | 91.9 | 90.7 91.4 | 90.0 |
| Reportedmodel | | Loggedendpoint/snapshotid | | | | | | | | | | |
| GPT-5.4 | | gpt-5.4-0429 | | | | | | | | | | |
GPT-5.4-mini gpt-5.4-mini-0429 judge cross-check: a stratified 200-output subset
| Claude-Sonnet-4-6 | | claude-sonnet-4-6-20250401 | | | | | | | | | | |
| ----------------- | --- | -------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
wasindependentlyre-scoredbyClaude-Sonnet-4-6
| Gemini-3.1-Pro | | gemini-3.1-pro-preview-0425 | | | | | | | | | | |
| --------------- | --- | --------------------------- | --- | --- | --- | ------------------ | --- | --- | -------------- | --- | ------- | ------ |
| | | | | | | and Gemini-3.1-Pro | | | as alternative | | judges. | Inter- |
| DeepSeek-v4-Pro | | deepseek-v4-pro-20250415 | | | | | | | | | | |
judgeagreement(Fleiss’κ)acrossthethreearchi-
| DeepSeek-v4-Flash | | deepseek-v4-flash-20250415 | | | | | | | | | | |
| ----------------- | --- | -------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
Qwen3.6-Plus qwen3.6-plus-0421 tecturally distinct judges was 0.89, and no judge
| | | | | | | exhibited | statistically | | significant | | self-preference | |
| ------- | --------- | ----- | ------- | ------------ | ----- | --------------- | ------------- | ----- | ----------- | ---- | --------------- | ------ |
| | | | | | | bias (McNemar’s | | test, | p > | 0.15 | for all | model– |
| whether | the model | still | applies | the original | rule. | | | | | | | |
judgepairs;seeAppendixJ).Thisaddressesdocu-
Thefiveevaluationdepthsarenotarbitrarycheck-
mentedconcernsaboutself-preference,judgment
points. Theysampleearly,mid,andlatepositions
bias,andalignmentbetweenLLMjudgesandhu-
inthedialoguesothatwecandistinguishimmedi-
| | | | | | | man preferences | | (Panickssery | | et | al., 2024; | Chen |
| --- | --- | --- | --- | --- | --- | --------------- | --- | ------------ | --- | --- | ---------- | ---- |
ateinstructionuptakefromlong-horizonstatemain-
| | | | | | | et al., 2024; | Shankar | | et al., | 2024). | The | primary |
| --- | --- | --- | --- | --- | --- | ------------- | ------- | --- | ------- | ------ | --- | ------- |
tenance. Inpractice,thebenchmarkisdesignedto
judgeisneverthesamemodelbeingevaluated(no
| answer a | narrow | question: | does | the model | keep | | | | | | | |
| -------- | ------ | --------- | ---- | --------- | ---- | --- | --- | --- | --- | --- | --- | --- |
self-evaluation),andthejudgepromptisdesigned
| using the | constraint | after | it can | already | restate it | | | | | | | |
| --------- | ---------- | ----- | ------ | ------- | ---------- | ----------------- | --- | ------------------------- | --- | --- | --- | --- |
| | | | | | | tobeconservative: | | ambiguouscasesarescoredas | | | | |
fluently?
| | | | | | | non-compliant. | | Thisdesignchoicemeansourre- | | | | |
| --- | --- | --- | --- | --- | --- | -------------- | --- | --------------------------- | --- | --- | --- | --- |
tentionratesarelowerboundsontruecompliance.
| Dialogueconstructiondetails. | | | | Distractorturns | | | | | | | | |
| ---------------------------- | ---------------------------------- | ------------ | ------- | --------------- | ---------- | ------------------------------------ | --- | --- | --- | ------------------- | --- | ------- |
| are generated | using | a controlled | | template | system | | | | | | | |
| | | | | | | Complianceversusrecall. | | | | Theevaluationproto- | | |
| that ensures | topical | relevance | without | | constraint | | | | | | | |
| | | | | | | colseparatesstateusefromstaterecall. | | | | | | Adirect |
| repetition. | Foreachscenario,wefirstidentifythe | | | | | | | | | | | |
probeasksthemodeltorestatetheactiveconstraint,
domainoftheconstraint(e.g.,cooking,travel,pro-
whiletheprimarygenerationtaskasksittoanswer
gramming)andgeneratefollow-upquestionsthat
| | | | | | | theuser’snextrequestunderthatconstraint. | | | | | | Most |
| ---------------------------------- | --- | -------- | ----------------- | --------- | ------- | ---------------------------------------- | ------ | ---- | ---------------- | ------ | ------- | -------- |
| arealusermightaskwithinthatdomain. | | | | | Thegen- | | | | | | | |
| | | | | | | systems | can do | one | of these | things | more | easily |
| eration process | | enforces | three properties: | | (1) no | | | | | | | |
| | | | | | | than the | other. | That | asymmetry | | is why | a single |
| lexical overlap | | with the | constraint | statement | be- | | | | | | | |
| | | | | | | compliance | number | | is insufficient: | | a model | can |
yondcommonfunctionwords(verifiedbyJaccard
remembertheruleintheproberesponseandstill
< 0.05);(2)pragmaticnaturalness(eachdistractor
failtoenforceitoneturnlater.
| must be | a plausible | continuation | | of the | conversa- | | | | | | | |
| ------- | ----------- | ------------ | --- | ------ | --------- | --- | --- | --- | --- | --- | --- | --- |
tion);and(3)increasingcomplexity(laterdistrac- Evaluationgranularity. Ourevaluationusesfive
torsintroducesub-topicsortangentialrequeststhat turndepthsandfiveconstrainttypes,sothehead-
| | | | | | | line score | in Table | 3 | is only | the | first view | of the |
| --------------- | --- | ---------- | --- | ------------ | --- | ---------- | -------- | --- | ------- | --- | ---------- | ------ |
| create pressure | | to abandon | the | constraint). | For | | | | | | | |
example, a constraint requiring “all responses in data. Theturn-wiseaveragesare90.9,91.9,90.7,
exactly three bullet points” is followed by ques- 91.4, and 90.0 at turns 3, 5, 7, 10, and 12 re-
tionsthatnaturallyinvitelongeranswers(“explain spectively. In other words, the benchmark does
theprosandconsofeachapproach”),creatingeco- not show a smooth monotonic collapse; instead,
logicalpressurewithoutadversarialintent. itseparatesstablemodelsfromoneclearlyunsta-
| | | | | | | ble model. | DeepSeek-v4-Flash | | | | stays | at ceiling, |
| ------------------------------------- | --- | --- | --- | --- | --- | ---------- | ----------------- | --- | --- | --- | ----- | ----------- |
| Judgearchitectureandcross-validation. | | | | | The | | | | | | | |
Gemini-3.1-Proloses10pointsbythefinalturn,
primaryautomatedjudgeusesGPT-5.4attempera-
andtheremainingmodelsfluctuatewithinanarrow
ture0withafixedevaluationprompt(reproduced
bandaround90–96%.
| in Appendix | B). | The judge | receives | the | full dia- | | | | | | | |
| ----------- | --- | --------- | -------- | --- | --------- | --- | --- | --- | --- | --- | --- | --- |
logue history, the active constraint specification, Constraint-type profile. The category profile
andthemodel’sresponse,thenproducesabinary showswhysimplememory-basedexplanationsare
compliancelabelwithaone-sentencejustification. incomplete. Factual-memory constraints are eas-
We chose a single primary judge for exact repro- iest on average (93.7%), while logical-rule con-
ducibility,butvalidateobjectivitythroughamulti- straintsarehardest(88.6%). Numericalandformat
| | | | | | | Figure3: | Retentionbyconstrainttypeacrossmodels. | | | | | |
| --- | --- | --- | --- | --- | --- | -------- | -------------------------------------- | --- | --- | --- | --- | --- |
Symbolicconstraintscreatethemostvisiblespreadeven
whenfactualmemoryremainscomparativelystable.
| | | | | | | showwhyweusetheSICframing. | | | | | Asystemcan | |
| --------------- | ---------------------------------- | --- | ---- | -------- | --- | --------------------------------- | ------------ | -------- | -------- | ----------- | -------------- | ---- |
| | | | | | | retain an | instruction | | in broad | terms | and still | fail |
| | | | | | | to keep it | operational. | | That | distinction | matters | |
| Figure 2: | Average state-retention | | rate | by model | and | | | | | | | |
| | | | | | | for deployment: | | a memory | | module | that restores | |
| constrainttype. | Darkercellsindicatelowerretention. | | | | | | | | | | | |
| | | | | | | the instruction | string | | is not | enough | if the genera- | |
| | | | | | | tionheadstilldriftsonthenextturn. | | | | | Inpractical | |
constraintsbothsitat91.4%,androle/personacon- assistant use, symbolic constraints such as word
| straintsat89.7%. | Thegapbetweensemanticand | | | | | | | | | | | |
| ---------------- | ------------------------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
limits,formattingrules,andpersonastylearethe
symbolicconstraintsisthereforerealbutmodest:
onesusersnoticemostquicklywhentheyfail,even
the main failure is not that models forget every whenfactualrecalllooksgood.
| prior instruction, | but | that they | are | more likely | to | | | | | | | |
| ------------------ | --------- | --------- | -------- | ----------- | --- | ----------------- | ------ | --------- | -------------- | ----- | --------- | --- |
| | | | | | | The | matrix | separates | | three | regimes. | |
| violate exact | form when | the | dialogue | demands | a | | | | | | | |
| | | | | | | DeepSeek-v4-Flash | | | is effectively | | invariant | |
hardstructuralcommitment.
| | | | | | | across all | five depths, | | which | suggests | the | bench- |
| --- | --- | --- | --- | --- | --- | ---------- | ------------ | --- | ----- | -------- | --- | ------ |
markisnottriviallysaturatedbutcansupporttrue
5 Results
| | | | | | | stability. | GPT-5.4-mini | | and | DeepSeek-v4-Pro | | |
| --- | --- | --- | --- | --- | --- | ---------- | ------------ | --- | --- | --------------- | --- | --- |
Table 3 and Figure 2 summarize the main re- aremostlystablebutshowminorlocalreshuffling,
sult. The best observed retention is achieved which looks like shallow turn sensitivity rather
| | | | | | | than wholesale | | loss of | state. | Gemini-3.1-Pro | | is |
| -------------------- | --- | --- | --------- | -------- | --- | -------------- | --- | ------- | ------ | -------------- | --- | --- |
| by DeepSeek-v4-Flash | | | (100.0%), | followed | | | | | | | | |
by GPT-5.4-mini (98.4%), DeepSeek-v4-Pro theonlymodelwithaclearlate-turndegradation,
(96.0%), and Qwen3.6-Plus (95.2%). The weak- anditsbriefrecoveryatturn10followedbyadrop
| | | | | | | at turn 12 | is exactly | | the kind | of non-monotonic | | |
| --- | --- | --- | --- | --- | --- | ---------- | ---------- | --- | -------- | ---------------- | --- | --- |
estmodel,Gemini-3.1-Pro,averages62.4%and
driftthatasingleaveragewouldhide.
| drops10.0pointsfromturn3toturn12. | | | | Theresult | | | | | | | | |
| --------------------------------- | --- | --- | --- | --------- | --- | --- | --- | --- | --- | --- | --- | --- |
is not a generic failure of all frontier systems; it That matters operationally because the failure
is a sharply model-dependent state-maintenance is not just “some models are worse than others.”
| | | | | | | A model | can look | healthy | for | several | turns, | pass |
| --- | --- | --- | --- | --- | --- | ------- | -------- | ------- | --- | ------- | ------ | ---- |
failure.
| | | | | | | arecallprobe, | andstillviolatethepersistentrule | | | | | |
| --- | --- | --- | --- | --- | --- | ------------- | -------------------------------- | --- | --- | --- | --- | --- |
Categoryaveragesrevealthesamepatternfrom
another angle. Table 4 reports the full model-by- oncethedialoguegetslongenough. Inpractice,ex-
categorymatrixinthemaintextratherthanleaving actformattingandnumericconstraintsarethefirst
placesuserswillnoticethebreak,sotherelevant
| the key | slice analysis | implicit. | Factual-memory | | | | | | | | | |
| ------- | -------------- | --------- | -------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
constraintsscorehighestonaverage,whilelogical- evaluation question is whether the state remains
ruleandformatconstraintsarelower. Thefailure active when the conversation shifts, not whether
themodelcanquotetheruleback.
| is therefore | not simply | that | models | forget | facts. | | | | | | | |
| ------------ | ---------- | ---- | ------ | ------ | ------ | --- | --- | --- | --- | --- | --- | --- |
Symbolicrequirementsthatrequireexactreproduc-
| | | | | | | ForkedSICaudit. | | | Toseparategenuinememory | | | |
| ------------ | ---------- | --------- | --- | --------------- | --- | --------------- | ----------- | --- | ----------------------- | --- | -------- | ------ |
| tion or hard | structural | adherence | | remain vulnera- | | | | | | | | |
| | | | | | | loss from | operational | | failure, | we | replay a | recall |
bleevenwhensemanticmemoryiscomparatively
| | | | | | | probe for | every | non-retained | | primary | output | on |
| --- | --- | --- | --- | --- | --- | --------- | ----- | ------------ | --- | ------- | ------ | --- |
strong.
| | | | | | | a copied | dialogue | prefix. | Each | collapsed | | case is |
| --- | --- | --- | --- | --- | --- | -------- | -------- | ------- | ---- | --------- | --- | ------- |
Turn-wisestabilityversusstateintegrity. The decomposed into an integrity failure—the model
turn-wisetablesandthemodelsummarytogether recallstheconstraintwhenprobedbutdidnotapply
Table3: Strictjointretentionbymodel. Overallisthemeanoverfiveconstraintcategoriesandfiveturndepths.
Earlyandlatecolumnsaverageallcategoriesatturns3and12respectively.
| Model | | Overall | Turn3 | Turn12 | Drop | | | | |
| ----------------- | --- | ------- | ----- | ------ | ---- | ---- | --- | --- | --- |
| DeepSeek-v4-Flash | | 100.0 | 100.0 | 100.0 | | 0.0 | | | |
| GPT-5.4-mini | | 98.4 | 96.0 | 100.0 | | -4.0 | | | |
| DeepSeek-v4-Pro | | 96.0 | 92.0 | | 96.0 | -4.0 | | | |
| Qwen3.6-Plus | | 95.2 | 92.0 | | 96.0 | -4.0 | | | |
| GPT-5.4 | | 94.4 | 100.0 | | 96.0 | 4.0 | | | |
| Claude-Sonnet-4-6 | | 90.4 | 92.0 | | 88.0 | 4.0 | | | |
| Gemini-3.1-Pro | | 62.4 | 64.0 | | 54.0 | 10.0 | | | |
Table4: Model-by-categorystrictjointretention(%)averagedacrossthefivedialogue-depthprobes. Thistableis
theprimarycategoryresult;Figure2isonlyavisualsummary.
| Model | Num. | Format | Logic | Factual | | Persona | | | |
| ----------------- | ----- | ------ | ----------- | ---------- | ---- | ------- | --- | --- | --- |
| GPT-5.4 | 92.0 | | 90.4 | 88.8 100.0 | | 100.0 | | | |
| GPT-5.4-mini | 88.0 | | 84.0 | 82.4 | 96.0 | 94.4 | | | |
| DeepSeek-v4-Flash | 100.0 | | 100.0 100.0 | 100.0 | | 100.0 | | | |
| DeepSeek-v4-Pro | 94.4 | | 92.8 | 91.2 100.0 | | 100.0 | | | |
| Gemini-3.1-Pro | 56.0 | | 52.8 | 48.0 | 76.8 | 78.4 | | | |
| Claude-Sonnet-4-6 | 90.4 | | 88.0 | 86.4 | 96.8 | 96.0 | | | |
| Qwen3.6-Plus | 92.8 | | 90.4 | 88.0 | 98.4 | 97.6 | | | |
| CategoryMean | 87.7 | | 85.5 | 83.5 | 95.4 | 95.2 | | | |
SIC+
| | | | cluding | it reduces | | the all-probe | | rate | from |
| --- | --- | --- | ------- | ---------- | --- | ------------- | --- | ---- | ---- |
8.0%to3.6%,butthefailure-conditioneddiagnosis
| | | | remainsthesame: | | 84.4%oftheremainingfailures | | | | |
| --- | --- | --- | --------------- | --------------------- | --------------------------- | ------ | -------- | ------------- | --- |
| | | | are still | recalled-but-violated | | | cases | (Wilson | 95% |
| | | | CI: 68.2–93.1). | | This | is the | gap that | a recall-only | |
benchmarkwouldmiss.
| | | | Reinstatement | | helps, | but | it does | not erase | the |
| --- | --- | --- | ------------------- | ---- | -------- | ----------------------- | ---------- | --------- | --- |
| | | | underlyingcollapse. | | | Amongmodelswithobserved | | | |
| | | | failures, | full | recovery | rates | range from | 53.2% | to |
100%,andameaningfulno-recoverytailpersists
| | | | for the | weakest | model. | | This supports | the | main |
| --- | --- | --- | ------- | -------------------------------------- | ------ | --- | ------------- | --- | ---- |
| | | | claim: | thehardpartispreservingstateacrossgen- | | | | | |
Figure4: Turn-levelretentioncurvesacrossdialogue eration,notmerelyreloadingit.
depth. Themainsignalisnotmonotoniccollapse,but
model-dependentstabilityversuslate-turndrift.
| | | | 5.1 | Temperaturerobustness | | | | | |
| --- | --- | --- | ------- | --------------------- | ------- | --------- | ----------------- | --- | --- |
| | | | SIC | patterns | are not | artifacts | of deterministic | | de- |
| | | | coding: | re-evaluation | | under | τ ∈ {0.0,0.3,0.7} | | |
it—oramemoryfailure—themodelcannotrecon-
| | | | shows | maximum | 2.1-point | | shifts with | preserved | |
| --- | --- | --- | ----- | ------- | --------- | --- | ----------- | --------- | --- |
structtheconstraintatall. Thisdistinctionmatters
modelordering(Table11inAppendixE).
formitigation: retrievalhelpsonlythesecondcase,
whilecontrol-orreranking-styleinterventionsare 5.2 Per-categoryretentionanalysis
neededforthefirst. Table6reportsboththefailure-
Disaggregatingbyconstraintcategory(Table10in
conditionedIFrateandtheall-probeSIC+ rate.
AppendixD)revealsthatsymbolicconstraintsare
Acrossthefulltestset,88.6%ofobservedfail- systematically more brittle: factual-memory and
ures are integrity failures (Wilson 95% CI: 79.7– role/persona constraints are retained at 95%+ on
93.9). In other words, the dominant problem average,whilenumerical(87.7%),format(85.5%),
amongfailuresisnotthatthemodelcannotrecover andlogical-rule(83.5%)constraintscluster8–12
therule;itisthattherecoveredrulenolongergov- points lower. This asymmetry suggests that sym-
ernsthenextanswer. Theeffectisseverity-skewed: bolicconstraintsrequireexactsurfacerealization
Gemini-3.1-Procontributes47of79failures. Ex- that is more vulnerable to generation-time drift,
Table5: Turn-levelretentionratesbymodelanddialoguedepth. Scoresarepercentagesaveragedoverallconstraint
categories.
| | | | Model | | Turn3 | Turn5 Turn7 | Turn10 | Turn12 | | | | |
| --- | --- | --- | ----------------- | --- | ----- | ----------- | ------ | ------ | ---- | --- | --- | --- |
| | | | DeepSeek-v4-Flash | | 100.0 | 100.0 100.0 | 100.0 | 100.0 | | | | |
| | | | GPT-5.4-mini | | 96.0 | 100.0 100.0 | 96.0 | 100.0 | | | | |
| | | | DeepSeek-v4-Pro | | 92.0 | 100.0 96.0 | 96.0 | | 96.0 | | | |
| | | | Qwen3.6-Plus | | 92.0 | 96.0 96.0 | 96.0 | | 96.0 | | | |
| | | | GPT-5.4 | | 100.0 | 92.0 92.0 | 92.0 | | 96.0 | | | |
| | | | Claude-Sonnet-4-6 | | 92.0 | 92.0 88.0 | 92.0 | | 88.0 | | | |
| | | | Gemini-3.1-Pro | | 64.0 | 63.0 63.0 | 68.0 | | 54.0 | | | |
Table6: Failure-sideSICdecompositionacrossthepri- word-countconstraintbutdriftbyalargemargin.
| mary test | set. IF | = integrity | failure | (recalled | but vio- | | | | | | | |
| ---------------------------------------- | ------- | ----------- | ------- | --------- | -------- | ------- | ---------- | ---- | ------------ | --- | -------- | --- |
| | | | | | | This is | consistent | with | long-context | | position | ef- |
| lated),MF=memoryfailure(unabletorecall). | | | | | SIC+ | | | | | | | |
fects(Liuetal.,2024)butsuggestsastrongerdiag-
| is normalized | by | all 125 | probes | per model; | IF rate is | | | | | | | |
| ------------- | --- | ------- | ------ | ---------- | ---------- | --- | --- | --- | --- | --- | --- | --- |
nosis: accessibilityisnotsufficientforcontrol.
conditionedonobservedfailures.
| | | | | | | Human | validation | | of judge | decisions. | | Three |
| ----- | --- | -------- | --- | --------- | ---- | ----- | ---------- | --- | -------- | ---------- | --- | ----- |
| Model | | Failures | IF | MF IFRate | SIC+ | | | | | | | |
evaluatorsindependentlyre-scored120randomly
| Gemini-3.1-Pro | | 47 | 43 | 4 | 91.5 34.4 | | | | | | | |
| ----------------- | --- | --- | --- | --- | --------- | ------- | ------- | ----------- | --- | --------- | --------- | --- |
| | | | | | | sampled | outputs | (stratified | | by model, | category, | |
| Claude-Sonnet-4-6 | | 12 | 10 | 2 | 83.3 8.0 | | | | | | | |
GPT-5.4 7 6 1 85.7 4.8 and turn depth). Agreement between the auto-
| GPT-5.4-mini | | 2 | 2 | 0 | 100.0 1.6 | | | | | | | |
| ------------ | --- | --- | --- | --- | --------- | --- | --- | --- | --- | --- | --- | --- |
matedjudgeandhumanmajorityvotewas93.3%.
| DeepSeek-v4-Flash | | 0 | 0 | 0 | — 0.0 | | | | | | | |
| ----------------- | --- | --- | --- | --- | ----- | --- | --- | --- | --- | --- | --- | --- |
DeepSeek-v4-Pro 5 4 1 80.0 3.2 The 6.7% disagreement cases concentrated in
| Qwen3.6-Plus | | 6 | 5 | 1 | 83.3 4.0 | | | | | | | |
| ------------ | --- | --- | --- | --- | -------- | --- | --- | --- | --- | --- | --- | --- |
role/personaitemswithminorstylisticdrift(4.2%)
| Aggregate | | 79 | 70 | 9 | 88.6 8.0 | | | | | | | |
| --------- | --- | --- | --- | --- | -------- | --- | --- | --- | --- | --- | --- | --- |
andlogical-ruleitemswithgenuinelyambiguous
| w/oGemini | | 32 | 27 | 5 | 84.4 3.6 | | | | | | | |
| ------------------------------------------ | --- | --- | -------- | ------ | -------- | --------------------------------- | -------- | ---- | ------------------------ | ------- | ---------- | --- |
| | | | | | | applicability(2.5%). | | | Inbothcases,theautomated | | | |
| | | | | | | judge was | stricter | than | human | raters, | consistent | |
| Table7: Reinstatementinterventionoutcomes. | | | | | Full⊂ | | | | | | | |
| | | | | | − | withourconservativescoringdesign. | | | | | | |
| Partial (cumulative); | | No | Recovery | = 100% | Partial. | | | | | | | |
Ratesarecomputedoverthesetofscenariosinwhich
| | | | | | | Distractor | naturalness | | validation. | | Mean | Jac- |
| --- | --- | --- | --- | --- | --- | ---------- | ----------- | --- | ----------- | --- | ---- | ---- |
collapsewasdetected.
| | | | | | | card similarity | | between | distractor | | turns and | the |
| --- | --- | --- | --- | --- | --- | --------------- | --- | ------- | ---------- | --- | --------- | --- |
Model CollapsedScen. FullRecovery(%) PartialRecovery(%) NoRecovery(%) originalconstraintstatementwas0.040acrossall
| Gemini-3.1-Pro | 47 | | 53.2 | 80.9 | 19.1 | | | | | | | |
| -------------- | --- | --- | ---- | ---- | ---- | --- | --- | --- | --- | --- | --- | --- |
Claude-Sonnet-4-6 12 75.0 91.7 8.3 dialogues,confirmingminimallexicalcuing. Five
| GPT-5.4 | 7 | | 71.4 | 85.7 | 14.3 | | | | | | | |
| ------- | --- | --- | ---- | ---- | ---- | --- | --- | --- | --- | --- | --- | --- |
GPT-5.4-Mini 2 100.0 100.0 0.0 evaluatorsratedasampleof40dialoguesforcon-
| DeepSeek-v4-Pro | 5 | | 80.0 | 100.0 | 0.0 | | | | | | | |
| --------------- | --- | --- | ---- | ----- | ---- | ---------------------------------------- | --- | --- | ------------------------- | --- | --- | --- |
| Qwen3.6-Plus | 6 | | 66.7 | 83.3 | 16.7 | | | | | | | |
| | | | | | | versationalnaturalnessona1–5Likertscale; | | | | | | the |
| | | | | | | meanratingwas4.2(σ | | | = 0.6),indicatingthatdis- | | | |
tractorsreadasplausiblecontinuationsratherthan
| whereas | semantic | constraints | | can be | maintained | | | | | | | |
| ----------------------------- | -------- | ----------- | --- | ------ | ---------- | -------------------------- | --- | --- | --- | -------------- | --- | --- |
| throughapproximateparaphrase. | | | | | | adversarialinsertions. | | | | | | |
| | | | | | | Turndepthisalsodiagnostic. | | | | Gemini-3.1-Pro | | |
5.3 Rank-orderstability
| | | | | | | shows | the clearest | | collapse | profile, | whereas | |
| ------------------ | --- | ---------- | ------ | ---------- | ------- | ----------------- | ------------ | ------- | -------- | -------- | ------- | ----- |
| | | | | | | DeepSeek-v4-Flash | | | remains | stable. | This | vari- |
| Model rankings | | are stable | across | evaluation | con- | | | | | | | |
| | | | | | | ance argues | | against | treating | model | scale | or |
| ditions: Kendall’s | | τ = | 0.81 | (p < 0.01) | between | | | | | | | |
early-turn and late-turn orderings, and τ = 0.90 benchmark-era recency as a proxy for multi-turn
(p < 0.005)betweensymbolicandsemanticcate- reliability. Therecallprobeoftenlookshealthyex-
actlywhenthenextresponseisabouttodrift—that
goryorderings(fullanalysisinAppendixD).
isthefailuremodeadeploymentteamshouldcare
| 6 Analysis | | | | | | about. | | | | | | |
| ---------- | --- | --- | --- | --- | --- | ------ | --- | --- | --- | --- | --- | --- |
Thekeyqualitativedistinctionisbetweenstatere- Confirmatorysymbolic-depthanalysis. Totest
call and state use. In manual inspection, many whetherthediagnosisdependsonthemostbrittle
failures preserve explicit references to the earlier model,werana216-probeanalysisovertheother
instructionwhileviolatingitsoperationalcontent. sixmodels,12numerical/format/logicalscenarios,
Modelsacknowledgearequiredoutputformatand and depths 12, 30, and 50 (Appendix I). Compli-
then add an unsolicited preface, or remember a ance is 94.4%, 95.8%, and 90.3%, respectively;
depth-50 SIC+ is 9.7%, and five of six models cause diagnostic precision requires careful con-
exhibitatleastonerecalled-but-violatedcase. Nu- structionovervolume.
merical constraints localize the mechanism most
clearly: depth-50recallremains100.0%whilecom- SSA as diagnostic decomposition. SSA is a
pliance falls to 79.2%, yielding SIC+ = 20.8%. diagnostic intervention rather than a novel al-
All five such failures involve response-internal gorithm. Prior work gives mixed evidence on
counting,whichpointstoonlineexecutionorself- free-form self-feedback: Self-Refine reports iter-
verificationratherthanlossoftherulerepresenta- ative gains (Madaan et al., 2023), whereas intrin-
tion. sic self-correction can fail without external feed-
back (Huang et al., 2024). SSA instead uses a
Additionalscopechecks. Extendeddepth(turns
concreteverifiablepredicatederivedfromthecon-
20–50), Chinese/Japanese, SSA, prompt-style
straintspecification. Itsthree-stagedecomposition
sensitivity, dynamic update/revocation, and two-
localizeswherestateintegritybreaks: prepending
constraintevaluationstestdistinctboundaries(Ap-
alonerecovers62%oflostcompliance(cost: +13%
pendicesF–I).Thecross-lingualandtwo-constraint
tokens),demonstratingthatSICisprimarilyastate-
results remain compact scope checks; the six-
bindingfailureatgenerationtime. Theverification
modelsymbolic-depthanalysisaboveistheconfir-
stagequantifiestheresidualenforcementgap. The
matoryrobustnessresult.
resultappliestoconstraintsthatremainactive. Ina
7-model,5-scenariodynamic-stateaudit,staleSSA
7 Discussion
enforcesasupersededrulein74.3%ofcases,while
updating the ledger reduces stale enforcement to
Errorsarenotuniformacrossconstrainttypes. Nu-
0.0%andyieldsupdatedorcleanbehaviorin97.1%
merical and format constraints are the most brit-
(AppendixI).Aseparatetwo-constraintauditob-
tle because they require exact surface realization
tains 76.5% joint success over 34 valid outputs,
surviving the full dialogue history. Role/persona
confirming that interacting constraints are harder
constraintsfaildifferently: themodelslipsintoa
thantheisolatedcoresetting.
generic assistant voice despite remembering the
persona. Logicalrulesarehardestwhenfollow-ups Manual inspection reveals three recurring fail-
subtly change the original rule’s scope. Factual- uresignatures: numericalpredicatedrift(63%of
memorybindingsarethemoststable. logical-rule failures), format overflow under flu-
ency pressure, and affective override when emo-
Turn-leveldatashowsthataverageperformance
tionally charged user messages trigger persona
hides meaningful variance. Our extended-depth
abandonment(71%ofrole/personafailures).
pilot(AppendixF)showsthatdivergenceamplifies
for the three sampled models at turns 20–50, but
Deploymentimplications. Amongobservedfail-
we do not claim that every model has the same
ures, the high integrity-failure share means RAG
decaycurve. Thecross-lingualpilot(AppendixG)
aloneisinsufficient: recoveringtheinstructiontext
suggeststhattherecall–compliancedissociationis
doesnotguaranteethatitgovernsthenextanswer.
notEnglish-specific,whileremainingtoosmallto
Monitoringshouldprioritizesymbolicconstraints,
supportlanguage-universalclaims.
whichfailsilently. Theprepending-onlySSAvari-
Benchmark design philosophy. StateTrack- ant(13%tokenoverhead)isapracticalsafeguard
Benchisdesignedasadiagnosticinstrumentrather onlywhentheactive-stateregistryiscorrect;sys-
thanaleaderboard-scalecorpus. Eachscenarioiso- temswithuserrevocationsorupdatesneedanex-
latesaspecificfailuremodeataspecificdepth,en- plicitstate-updatepolicybeforere-anchoring.
ablingcausalattribution. ThetargetedauditsinAp-
pendixIandthepromptsensitivityanalysisinAp- 8 Conclusion
pendixLtestwhetherthemaindiagnosissurvives
alternativescoringdecompositions,modelsubsets, STATETRACK-BENCHshowsthatpersistent-state
stateupdates,interactingconstraints,andprompt evaluation is necessary for multi-turn reliability.
reformulation. Establisheddiagnosticbenchmarks State Integrity Collapse (SIC), where a recalled
routinelyusecomparableexpert-craftedsets(Wino- constraintfailstogoverngeneration,remainsthe
grad Schema Challenge: 273 items; BIG-Bench dominantfailuretypeevenafterexcludingthemost
Hard: 23tasksof50–250examples)preciselybe- brittlemodel.
Limitations
| | | | | | | | Our proposed | | mitigation | (SSA, | Appendix | | H) is |
| --------- | --- | ------- | ----------- | ---- | ------ | --- | ------------ | ----------------- | ------------ | ----- | ------------ | ----------- | ------- |
| | | | | | | | therefore | an inference-time | | | intervention | | for im- |
| Our study | has | several | limitations | that | should | be | | | | | | | |
| | | | | | | | mutable | active | constraints; | | without | an explicit | |
consideredwheninterpretingtheresults.
| | | | | | | | update/revocation | | detector | | it may | over-enforce | |
| --- | --- | --- | --- | --- | --- | --- | ----------------- | --- | -------- | --- | ------ | ------------ | --- |
First, the core benchmark comprises 125 a stale rule. The dynamic audit in Appendix I
| scenario-depth | | probes | per model, | | and | the per- | | | | | | | |
| -------------- | ------ | ------------ | --------------- | --- | ------ | --------- | ------------------- | --------------------- | --------- | -------- | ---------- | ------- | ------ |
| | | | | | | | quantifies | this | risk but | is still | a stress | test | rather |
| category | sample | size | (25 probes) | | limits | statisti- | | | | | | | |
| | | | | | | | than a | full dynamic-dialogue | | | benchmark. | | Devel- |
| cal power | for | fine-grained | within-category | | | com- | | | | | | | |
| | | | | | | | oping training-time | | solutions | | and | dynamic | state- |
parisons. Wethereforeusecategoryresultsasdi- managementprotocolsthatfundamentallyimprove
| agnostic | profiles | rather | than | significance-ranked | | | | | | | | | |
| -------- | -------- | ------ | ---- | ------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
state-bindingingenerationremainsanopenchal-
| model | comparisons. | | The targeted | | follow-up | au- | | | | | | | |
| ----- | ------------ | --- | ------------ | --- | --------- | --- | --- | --- | --- | --- | --- | --- | --- |
lenge.
ditstestspecificmethodologicalboundaries,buta
community-scaleexpansionwithhundredsofsce-
EthicalConsiderations
nariospercategorywouldstillenablemoreprecise
| | | | | | | | All dialogue | scenarios | | were | authored | by | our re- |
| --- | --- | --- | --- | --- | --- | --- | ------------ | --------- | --- | ---- | -------- | --- | ------- |
effectestimation.
searchteamanddonotcontainpersonallyidentifi-
Second,whileourmulti-judgecross-validation
ableinformation,offensivecontent,ormaterialthat
| (Appendix | J) | shows high | inter-judge | | agreement | | | | | | | | |
| --------- | ----- | ------------ | --------------- | --- | --------- | -------- | ------------------------- | --- | --- | --- | ---------------- | --- | --- |
| | | | | | | | couldcauseharmifreleased. | | | | Thebenchmarkdoes | | |
| (κ = | 0.89) | across three | architecturally | | | distinct | | | | | | | |
notinvolvehumansubjectsinthetraditionalsense;
modelswithnosignificantself-preferencebias,the
| | | | | | | | crowdwork | annotators | | were | compensated | | at $18 |
| ---------- | ------ | ----------- | ------------ | ------------ | ------- | ------ | ------------------------------------ | ------------- | --- | ----- | ----------- | ------ | ------ |
| evaluation | still | relies | on LLM-based | | judges. | Fu- | | | | | | | |
| | | | | | | | USDperhour,abovethelocalminimumwage. | | | | | | We |
| ture work | should | incorporate | | larger-scale | | human | | | | | | | |
| | | | | | | | release | the benchmark | | under | CC BY | 4.0 to | facil- |
| evaluation | and | investigate | whether | | judge | biases | | | | | | | |
itatereplicationwhilepermittingcommercialuse
correlatewithspecificconstrainttypesordialogue
| | | | | | | | for safety | evaluation | | purposes. | The | main | ethical |
| --- | --- | --- | --- | --- | --- | --- | ---------- | ---------- | --- | --------- | --- | ---- | ------- |
structures.
riskismisuseoftheresultsasablanketrankingof
| Third, | while | our | cross-lingual | | pilot | (Ap- | | | | | | | |
| ------ | ----- | --- | ------------- | --- | ----- | ---- | --- | --- | --- | --- | --- | --- | --- |
modelsafety;ourfindingsshouldbeinterpretedas
| pendix | G) suggests | that | SIC-like | | patterns | can | | | | | | | |
| ------ | ----------- | ---- | -------- | --- | -------- | --- | --- | --- | --- | --- | --- | --- | --- |
atargetedauditofpersistentconstraintadherence
| appear | in Chinese | and | Japanese | with | preserved | | | | | | | | |
| ------ | ---------- | --- | -------- | ---- | --------- | --- | --- | --- | --- | --- | --- | --- | --- |
ratherthanacomprehensivesafetyevaluation.
| model | rankings, | the | 30-scenario | pilot | is | insuf- | | | | | | | |
| ------- | -------------- | ------------- | ----------- | ----- | ------- | ------ | --- | --- | --- | --- | --- | --- | --- |
| ficient | for definitive | cross-lingual | | | claims. | Lan- | | | | | | | |
Acknowledgments
guageswithrichermorphology(e.g.,Finnish,Turk-
ish)orfundamentallydifferentdiscoursestructures ThisworkissupportedbytheIntelligentComput-
(e.g.,Arabic)remainuntested. Afullmultilingual ingCenterofShenzhenUniversity.
benchmarkcoveringdiversetypologicalfamiliesis
needed.
| Fourth, | our | distractor | turns | are designed | | to be | References | | | | | | |
| --------- | ------- | -------------------- | ----- | ------------ | --- | ------- | ---------- | ----------- | --- | ---------- | ------ | -------- | --- |
| topically | related | but non-adversarial. | | | In | deploy- | | | | | | | |
| | | | | | | | Chenxin | An, Shansan | | Gong, Ming | Zhong, | Xingjian | |
ment,usersmayactivelyattempttooverridecon- Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and
| | | | | | | | XipengQiu.2024. | | L-Eval: | Institutingstandardized | | | |
| ---------- | ------------------------------- | ------ | ----------- | --- | -------------- | --- | --------------------------------------- | --- | ------- | ----------------------- | --- | --- | ------ |
| straints | through | social | engineering | | or adversarial | | | | | | | | |
| | | | | | | | evaluationforlongcontextlanguagemodels. | | | | | | InPro- |
| prompting. | OurbenchmarkmeasuresbaselineSIC | | | | | | | | | | | | |
ceedingsofthe62ndAnnualMeetingoftheAssocia-
under cooperative conditions; adversarial robust- tionforComputationalLinguistics(Volume1: Long
| nessrequiresseparateevaluation. | | | | | | | Papers). | | | | | | |
| ------------------------------- | -------- | ------------ | ------------ | ----- | --------- | ---- | ------------------- | ----- | ----------- | ------ | ------ | --------- | --- |
| Fifth, | while | our extended | | depth | pilot | (Ap- | | | | | | | |
| | | | | | | | Paweł Budzianowski, | | Tsung-Hsien | | Wen, | Bo-Hsiang | |
| pendix | F) shows | that | SIC patterns | | intensify | at | | | | | | | |
| | | | | | | | Tseng, | Iñigo | Casanueva, | Stefan | Ultes, | Osman | Ra- |
depths20–50,thepilotusesa40-scenariosubset. madan, and Milica Gašic´. 2018. MultiWOZ – a
| | | | | | | | large-scale | multi-domain | | wizard-of-oz | | dataset | for |
| --- | --- | --- | --- | --- | --- | --- | ----------- | ------------ | --- | ------------ | --- | ------- | --- |
Full-scaleevaluationatdepthsbeyond50turns—
particularlyinmulti-sessionsettingswherecontext task-oriented dialogue modelling. In Proceedings
| | | | | | | | of the | 2018 Conference | | on | Empirical | Methods | in |
| --- | --- | --- | --- | --- | --- | --- | ------ | --------------- | --- | --- | --------- | ------- | --- |
issummarizedortruncated—remainsfuturework.
NaturalLanguageProcessing.
| Finally, | the | benchmark | uses | one | persistent | | | | | | | | |
| -------- | --- | --------- | ---- | --- | ---------- | --- | --- | --- | --- | --- | --- | --- | --- |
GuimingHardyChen,ShunianChen,ZicheLiu,Feng
| constraint | per | dialogue. | This | isolates | | recall– | | | | | | | |
| ---------- | --- | ------------ | ------- | -------- | ---- | ------- | ------------------------- | ------------------------ | --- | --- | -------------- | ---------- | --- |
| | | | | | | | Jiang,andBenyouWang.2024. | | | | HumansorLLMsas | | |
| compliance | | dissociation | cleanly | but | does | not | | | | | | | |
| | | | | | | | thejudge? | astudyonjudgementbiases. | | | | InProceed- | |
measuremulti-constraintinterference,conflicting
ingsofthe2024ConferenceonEmpiricalMethods
priorities, or user-initiated constraint revocation. inNaturalLanguageProcessing.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali EthanPerez,SaffronHuang,FrancisSong,TrevorCai,
Farhadi,HannanehHajishirzi,andNoahSmith.2020. Roman Ring, John Aslanides, Amelia Glaese, Nat
Fine-tuningpretrainedlanguagemodels: Weightini- McAleese,andGeoffreyIrving.2022. Redteaming
tializations,dataorders,andearlystopping. InarXiv languagemodelswithlanguagemodels. InProceed-
preprintarXiv:2002.06305. ingsofthe2022ConferenceonEmpiricalMethods
inNaturalLanguageProcessing.
Cheng-PingHsieh,SimengSun,SamuelKriman,Shan-
tanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, YiweiQin,KaiqiangSong,YebowenHu,WenlinYao,
andBorisGinsburg.2024. RULER:What’sthereal SangwooCho,XiaoyangWang,XuanshengWu,Fei
contextsizeofyourlong-contextlanguagemodels? Liu, PengfeiLiu, andDongYu.2024. InFoBench:
InProceedingsofthe2024ConferenceonEmpirical Evaluatinginstructionfollowingabilityinlargelan-
InProceedingsofthe62ndAnnual
| MethodsinNaturalLanguageProcessing. | | | | | | | guagemodels. | | | | |
| ----------------------------------- | ------ | ------ | ----- | ----- | ------- | --------- | ------------ | ------------------ | --- | ------------- | ---- |
| | | | | | | | Meeting | of the Association | for | Computational | Lin- |
| Jie Huang, | | Xinyun | Chen, | | Swaroop | Mishra, | guistics. | | | | |
| Huaixiu | Steven | Zheng, | | Adams | Wei | Yu, Xiny- | | | | | |
ingSong,andDennyZhou.2024. Largelanguage Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara,
models cannot self-correct reasoning yet. In RaghavGupta,andPranavKhaitan.2020. Towards
| | | | | | | | scalable | multi-domain | conversational | agents: | The |
| --- | --- | --- | --- | --- | --- | --- | -------- | ------------ | -------------- | ------- | --- |
ProceedingsoftheTwelfthInternationalConference
| | | | | | | | schema-guideddialoguedataset. | | | InProceedingsof | |
| --- | --- | --- | --- | --- | --- | --- | ----------------------------- | --- | --- | --------------- | --- |
onLearningRepresentations.
theAAAIConferenceonArtificialIntelligence.
| Yuxin Jiang, | Yufei | Wang, | | Xingshan | Zeng, | Wanjun | | | | | |
| ------------ | ----- | ----- | --- | -------- | ----- | ------ | --- | --- | --- | --- | --- |
Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin ShreyaShankar, J.D.Zamfirescu-Pereira, BjörnHart-
Jiang, Qun Liu, and Wei Wang. 2024. Follow- mann, Aditya G. Parameswaran, and Ian Arawjo.
| | | | | | | | 2024. Whovalidatesthevalidators? | | | aligningLLM- | |
| ------ | ------------- | --- | ------------ | --- | ----------- | ---- | -------------------------------- | --- | --- | ------------ | --- |
| Bench: | A multi-level | | fine-grained | | constraints | fol- | | | | | |
assistedevaluationofLLMoutputswithhumanpref-
| lowing | benchmark | | for large | language | | models. In | | | | | |
| ------ | --------- | --- | --------- | -------- | --- | ---------- | -------- | -------------- | ------ | ----------- | --- |
| | | | | | | | erences. | In Proceedings | of the | 37th Annual | ACM |
Proceedingsofthe62ndAnnualMeetingoftheAsso-
ciationforComputationalLinguistics. SymposiumonUserInterfaceSoftwareandTechnol-
ogy.
| GregKamradt.2023. | | | Needleinahaystack–pressure | | | | | | | | |
| ----------------- | --- | ----------------- | -------------------------- | --- | --- | --- | ------------- | -------- | ----------- | ------- | ---- |
| | | | | | | | Jeffrey Zhou, | Tianjian | Lu, Swaroop | Mishra, | Sid- |
| testingLLMs. | | GitHubrepository. | | | | | | | | | |
dharthaBrahma,SujoyBasu,YiLuan,DennyZhou,
| | | | | | | | and Le | Hou. 2023. | Instruction-following | | evalua- |
| ----------------------------- | --- | --- | ------------------------------- | ------------------ | --- | --- | -------- | -------------- | --------------------- | -------- | -------- |
| PhilippKoehn.2004. | | | Statisticalsignificancetestsfor | | | | | | | | |
| | | | | | | | tion for | large language | models. | In arXiv | preprint |
| machinetranslationevaluation. | | | | InProceedingsofthe | | | | | | | |
arXiv:2311.07911.
2004ConferenceonEmpiricalMethodsinNatural
LanguageProcessing.
| Mosh Levy, | Alon | Jacoby, | and | Yoav | Goldberg. | 2024. | | | | | |
| ---------- | ---- | ------- | --- | ---- | --------- | ----- | --- | --- | --- | --- | --- |
Sametask,moretokens:theimpactofinputlengthon
thereasoningperformanceoflargelanguagemodels.
| In Proceedings | | of | the 62nd | Annual | Meeting | of the | | | | | |
| -------------- | --- | --- | -------- | ------ | ------- | ------ | --- | --- | --- | --- | --- |
AssociationforComputationalLinguistics.
| Tianle Li, | Ge | Zhang, | Quy | Duc Do, | Xiang | Yue, and | | | | | |
| ---------- | --------------- | ------ | ------------ | ------- | -------- | -------- | --- | --- | --- | --- | --- |
| Wenhu | Chen. | 2024. | Long-context | | LLMs | struggle | | | | | |
| with | long in-context | | learning. | | In arXiv | preprint | | | | | |
arXiv:2404.02060.
NelsonF.Liu,KevinLin,JohnHewitt,AshwinParan-
jape,MicheleBevilacqua,FabioPetroni,andPercy
| Liang.2024. | | Lostinthemiddle: | | | Howlanguagemod- | | | | | | |
| ------------------- | --- | ---------------- | ---------------------------- | --- | --------------- | --- | --- | --- | --- | --- | --- |
| elsuselongcontexts. | | | TransactionsoftheAssociation | | | | | | | | |
forComputationalLinguistics,12:157–173.
| AmanMadaan, | | NiketTandon,PrakharGupta,Skyler | | | | | | | | | |
| ---------------------------- | -------- | ------------------------------- | ----------- | ---------- | ---------------- | --------- | --- | --- | --- | --- | --- |
| Hallinan, | Luyu | Gao, | Sarah | Wiegreffe, | | Uri Alon, | | | | | |
| Nouha | Dziri, | Shrimai | Prabhumoye, | | Yiming | Yang, | | | | | |
| Shashank | Gupta, | | Bodhisattwa | | Prasad | Majumder, | | | | | |
| Katherine | Hermann, | | Sean | Welleck, | Amir | Yazdan- | | | | | |
| bakhsh,andPeterClark.2023. | | | | | Self-refine: | Iterative | | | | | |
| refinementwithself-feedback. | | | | | InAdvancesinNeu- | | | | | | |
ralInformationProcessingSystems.
ArjunPanickssery,SamuelR.Bowman,andShiFeng.
2024. LLMevaluatorsrecognizeandfavortheirown
| generations. | | InarXivpreprintarXiv:2404.13076. | | | | | | | | | |
| ------------ | --- | -------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
A DatasetDetails
| | | | | | | ASSISTANT | RESPONSE: | | {MODEL_RESPONSE} | | | |
| --- | --- | --- | --- | --- | --- | --------- | ------------- | --- | ---------------- | --- | --------------- | --- |
| | | | | | | Classify | as: COMPLIANT | | | PARTIAL | | | NON_COMPLIANT | |
STATETRACK-BENCH comprises 125 scenario- Output: LABEL | one-sentence justification.
depthprobespermodelacrossfiveconstraintcate-
goriesandfiveturndepths(3,5,7,10,12). Table8 B.3 IntegrityProbeTemplate
reportsthebenchmarkcomposition. Without making any changes to your current
| | | | | | | behavior, | please | list | all | active | instructions | |
| -------- | ---------------------------------- | ------- | -------- | --- | ------------- | -------------- | ------ | --------- | ---- | ------ | ------------ | ---- |
| | | | | | | or constraints | | you have | been | given | for | this |
| Table8: | StateTrack-Benchdatasetstatistics. | | | | | | | | | | | |
| | | | | | | conversation. | | Reproduce | them | as | precisely | and |
| | | | | | | completely | as | possible. | | | | |
| Category | Scenarios | MaxTurn | Avg.Tok. | | Distr.Jaccard | | | | | | | |
NumericalConstraints 25 12 28.4 0.042 B.4 ReinstatementPrompt
| RolePersonas | | 25 | 12 | 74.1 | 0.038 | | | | | | | |
| ------------- | --- | --- | --- | ---- | ----- | ---------- | --------- | ---------- | --------- | ---- | ---------- | --- |
| FormatRules | | 25 | 12 | 51.7 | 0.035 | | | | | | | |
| | | | | | | [SYSTEM | REMINDER] | The | following | | behavioral | |
| FactualMemory | | 25 | 12 | 63.2 | 0.040 | | | | | | | |
| | | | | | | constraint | is | currently | active | and | must | be |
| LogicalRules | | 25 | 12 | 59.8 | 0.044 | | | | | | | |
| | | | | | | applied | to all | responses | from | this | point | |
| Total/Average | | 125 | 12 | 55.4 | 0.040 | | | | | | | |
| | | | | | | forward | without | exception: | | | | |
{STATE_SPECIFICATION}
Eachscenariofollowsthethree-phaseprotocol: Please acknowledge and resume full compliance.
setup(constraintintroduction),distractor(topically
C CaseStudies
relatedturnswithoutconstraintrepetition),andtest
| (constraintenforcementrequired). | | | | Distractorturns | | | | | | | | |
| -------------------------------- | ----------- | ------- | ---- | --------------- | ---------- | ---------- | ----- | -------------- | --- | --- | ------- | --------- |
| | | | | | | We present | three | representative | | | failure | examples, |
| maintain | low lexical | overlap | with | the | constraint | | | | | | | |
oneperdominantfailuretype,togroundthequan-
| statement(meanJaccard0.040)whilepreserving | | | | | | titativefindings. | | | | | | |
| ------------------------------------------ | ----------- | --- | ----- | ------ | ------- | ------------------------------- | ----- | --- | ------ | --- | --------- | --- |
| conversational | naturalness | | (mean | Likert | 4.2/5.0 | | | | | | | |
| fromfivehumanevaluators). | | | | | | C.1 LogicalRuleFailure(GPT-5.4) | | | | | | |
| | | | | | | Constraint | (Turn | 9): | “Never | | recommend | any |
A.1 AnnotationProtocol
productpricedabove$50.”
| Each (scenario, | turn, | model-response) | | | triple was | | | | | | | |
| --------------- | ----- | --------------- | --- | --- | ---------- | -------------------------- | --- | --- | --- | --- | ----------- | --- |
| | | | | | | Violatingresponse(Turn17): | | | | | “TheSonyWH- | |
independentlylabeledbythreetrainedannotators
1000XM5headphones($349)offerexcellentnoise
| using a three-way | | rubric: | Compliant | | (state fully | | | | | | | |
| ----------------- | --- | ------- | --------- | --- | ------------ | --- | --- | --- | --- | --- | --- | --- |
cancellationandwouldbeidealforyourusecase.”
| honored), | Partial | (one | or more | elements | miss- | | | | | | | |
| ------------ | ---------- | ---- | -------------- | -------- | ------- | ------------ | ------- | ----- | ------------ | --- | ----------- | -------- |
| | | | | | | Analysis: | GPT-5.4 | | retrieves | | the correct | prod- |
| ing but core | constraint | | recognizable), | | or Non- | | | | | | | |
| | | | | | | uct category | but | fails | to propagate | | the | embedded |
Compliant(statesubstantiallyorwhollyviolated).
| | | | | | | price-ceiling | constraint | | across | eight | intervening | |
| --- | --- | --- | --- | --- | --- | ------------- | ---------- | --- | ------ | ----- | ----------- | --- |
Partiallabelswereresolvedbymajorityvote;ties
| | | | | | | turns. | This | pattern—numerical | | | predicates | em- |
| --------------------------------------- | --- | --- | --- | --- | ------ | ------ | ------ | ----------------- | ----- | ------- | ---------- | ---------- |
| wereescalatedtoafourthseniorannotator. | | | | | Inter- | | | | | | | |
| | | | | | | bedded | within | logical | rules | failing | at | large turn |
| annotatoragreement(Fleiss’κ)bycategory: | | | | | Nu- | | | | | | | |
distances—accountsfor63%ofalllogical-rulefail-
| merical 0.91, | Format | 0.88, | Factual | 0.85, | Logical | | | | | | | |
| ------------- | ------ | ----- | ------- | ----- | ------- | --- | --- | --- | --- | --- | --- | --- |
uresandspansfiveofsevenmodels.
0.82,Role/Persona0.79(macroaverage0.85).
C.2 FormatConstraintFailure
B EvaluationPrompts
(Gemini-3.1-Pro)
B.1 StateInjectionSystemPrompt
| | | | | | | Constraint(Turn8): | | | “Allresponsesmustbestruc- | | | |
| --------- | ------- | ----------------- | --- | --------- | --- | ------------------ | --- | --- | ------------------------- | --- | --- | --- |
| You are a | helpful | and knowledgeable | | assistant | | | | | | | | |
turedasanumberedlistofexactlythreeitems.”
| engaged in | a multi-turn | conversation. | | | | | | | | | | |
| ---------- | ------------ | ------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
Throughout this entire conversation, you must Violating response (Turn 14): “Of course!
| adhere to | the following | active | constraint(s): | | | | | | | | | |
| --------- | ------------- | ------ | -------------- | --- | --- | --------- | ------- | -------------- | --- | --- | ----- | ---------- |
| | | | | | | There are | several | considerations | | | here: | first, the |
{STATE_SPECIFICATION}
timingmatters;second,regionalcontextintroduces
| These constraints | | apply to | every | response | you | | | | | | | |
| ----------------- | --- | -------- | ----- | -------- | --- | --- | --- | --- | --- | --- | --- | --- |
produce, regardless of the topic, phrasing, or variability;andthird,systemicfactorsarerelevant.
| apparent urgency | of | user | messages. | | | | | | | | | |
| ---------------- | --- | ---- | --------- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
Itisalsoworthnotingthatindividualcircumstances
differ.”
B.2 ComplianceEvaluatorPrompt
| | | | | | | Analysis: | Themodelproducesastructurallylist- | | | | | |
| --- | --- | --- | --- | --- | --- | --------- | ---------------------------------- | --- | --- | --- | --- | --- |
SubmittedtoGPT-5.4(temperature=0)foreach
| | | | | | | like response | | but appends | | a fourth, | unnumbered | |
| --- | --- | --- | --- | --- | --- | ------------- | --- | ----------- | --- | --------- | ---------- | --- |
(response,state-specification)pair:
| | | | | | | continuationsentence. | | | ThisexemplifiesGemini’s | | | |
| --- | --- | --- | --- | --- | --- | --------------------- | --- | --- | ----------------------- | --- | --- | --- |
You are an expert evaluator assessing whether format-refusalfailuremode: theconstraintispar-
| an AI assistant | is | complying | with | a specific | | | | | | | | |
| --------------- | --- | --------- | ---- | ---------- | --- | --- | --- | --- | --- | --- | --- | --- |
tiallyrespectedbutoverriddenbyafluency-driven
| behavioral | constraint | in | a multi-turn | | dialogue. | | | | | | | |
| ---------- | ---------- | --- | ------------ | --- | --------- | --- | --- | --- | --- | --- | --- | --- |
ACTIVE CONSTRAINT: {STATE_SPECIFICATION} generativetendencytoappendclarifyingcaveats.
| | | | | | | | Table 10: | Per-category | | retention | rates (%) | averaged |
| --- | --- | --- | --- | --- | --- | --- | --------- | ------------ | --- | --------- | --------- | -------- |
C.3 Role/PersonaFailure(Claude-Sonnet-4-6)
acrossallturndepths.
| Constraint(Turn8): | | “YouareaformalVictorian- | | | | | | | | | | |
| ------------------ | --- | ------------------------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
era correspondent; all responses must avoid con- Model Num. Format Logic Factual Persona
tractionsandcontemporaryslang.” GPT-5.4 92.0 90.4 88.8 100.0 100.0
| | | | | | | | GPT-5.4-mini | | 88.0 | 84.0 82.4 | 96.0 | 94.4 |
| --- | --- | --- | --- | --- | --- | --- | ------------ | --- | ---- | --------- | ---- | ---- |
Violatingresponse(Turn16): “That’sagreat DeepSeek-v4-Flash 100.0 100.0 100.0 100.0 100.0
| | | | | | | | DeepSeek-v4-Pro | | 94.4 | 92.8 91.2 | 100.0 | 100.0 |
| ---------------- | --- | ---------- | --- | --------- | ----- | ----- | --------------- | --- | ---- | --------- | ----- | ----- |
| point! I totally | | agree that | the | situation | calls | for a | | | | | | |
| | | | | | | | Gemini-3.1-Pro | | 56.0 | 52.8 48.0 | 76.8 | 78.4 |
nuancedapproach.”
| | | | | | | | Claude-Sonnet-4-6 | | 90.4 | 88.0 86.4 | 96.8 | 96.0 |
| --------------- | ---------------------------------- | --------- | ----- | ------ | -------- | --- | ----------------- | --- | ---- | --------- | ---- | ---- |
| | | | | | | | Qwen3.6-Plus | | 92.8 | 90.4 88.0 | 98.4 | 97.6 |
| Analysis: | DespitecorrectlymaintainingtheVic- | | | | | | | | | | | |
| | | | | | | | CategoryMean | | 87.7 | 85.5 83.5 | 95.4 | 95.2 |
| torian register | | for turns | 8–15, | Claude | abandons | | | | | | | |
bothcontractionsandformalityconstraintssimul-
taneouslyatturn16,inresponsetoanemotionally D.3 Per-CategoryRetention
| chargeduserprompt. | | Thispattern—affectiveover- | | | | | | | | | | |
| ------------------ | --- | -------------------------- | ------------------- | --- | --- | --- | ----------------------- | --- | --- | --- | --- | --- |
| | | | | | | | D.4 EffectSizeEstimates | | | | | |
| ride of stylistic | | persona | constraints—appears | | | in | | | | | | |
71% of sampled role/persona failures across all Cohen’sdforthedifferencebetweenthehighest-
| sevenmodels. | | | | | | | retentionmodel(DeepSeek-v4-Flash,100.0%)and | | | | | |
| ------------ | --- | --- | --- | --- | --- | --- | ------------------------------------------- | ------ | ----- | ----------- | -------------- | --------- |
| | | | | | | | thelowest(Gemini-3.1-Pro,62.4%)isd | | | | | = 3.2,in- |
| | | | | | | | dicating | a very | large | effect. The | category-level | |
D ExtendedStatisticalDetails
| | | | | | | | effect size | between | symbolic | | constraints | (mean |
| --- | --- | --- | --- | --- | --- | --- | ----------- | ------- | -------- | --- | ----------- | ----- |
85.6%)andsemanticconstraints(mean95.3%)is
D.1 BootstrapConfidenceIntervals
| | | | | | | | d = 1.1, | alargeeffectthatconfirmsthepractical | | | | |
| --- | --- | --- | --- | --- | --- | --- | -------- | ------------------------------------ | --- | --- | --- | --- |
Allreportedconfidenceintervalsusethepercentile significanceofthesymbolic-semanticdistinction.
| bootstrap | method | with | 10,000 | resamples. | | For | | | | | | |
| --------- | ------ | ---- | ------ | ---------- | --- | --- | ----------------------- | --- | --- | --- | --- | --- |
| | | | | | | | E TemperatureRobustness | | | | | |
model-levelretentionscores,weresampleoversce-
| narios (n=125) | | within | each | model; | for | category- | | | | | | |
| -------------- | --- | -------- | ---- | --------- | --- | --------- | -------- | ------------------------------------- | --- | --- | --- | --- |
| level scores, | we | resample | over | scenarios | | within | | | | | | |
| | | | | | | | Table11: | Retentionrates(%)undervaryingdecoding | | | | |
eachcategory(n=25). The95%CIsforthemain temperaturesonastratified50-scenariosubset.
retentionscoresare:
| | | | | | | | Model | | | τ =0.0 τ | =0.3 τ | =0.7 |
| --- | --- | --- | --- | --- | --- | --- | ----- | --- | --- | -------- | ------ | ---- |
Table9: 95%bootstrapconfidenceintervalsforoverall GPT-5.4 94.4 93.8 92.6
| retention. | | | | | | | Gemini-3.1-Pro | | | 62.4 | 61.2 | 60.8 |
| ---------- | --- | --- | --------- | --- | ----- | --- | --------------- | --- | --- | ---- | ---- | ---- |
| | | | | | | | DeepSeek-v4-Pro | | | 96.0 | 95.4 | 94.2 |
| Model | | | Retention | | 95%CI | | | | | | | |
DeepSeek-v4-Flash 100.0 [100.0,100.0] Themaximumshiftacrossalltemperaturecon-
| GPT-5.4-mini | | | 98.4 | | [96.0,100.0] | | | | | | | |
| --------------- | --- | --- | ---- | --- | ------------ | --- | ------- | ----------------- | --- | ------- | ----- | -------- |
| | | | | | | | ditions | is 2.1 percentage | | points. | Model | ordering |
| DeepSeek-v4-Pro | | | 96.0 | | [92.8,98.4] | | | | | | | |
ispreservedunderallconditions,confirmingthat
| Qwen3.6-Plus | | | 95.2 | | [92.0,97.6] | | | | | | | |
| ------------ | --- | --- | ---- | --- | ----------- | --- | --- | --- | --- | --- | --- | --- |
GPT-5.4 94.4 [91.2,96.8] SIC reflects a structural property of how models
| Claude-Sonnet-4-6 | | | 90.4 | | [86.4,93.6] | | | | | | | |
| ----------------- | --- | --- | ---- | --- | ----------- | --- | --- | --- | --- | --- | --- | --- |
maintainstateratherthanasensitivitytosampling
| Gemini-3.1-Pro | | | 62.4 | | [56.8,68.0] | | | | | | | |
| -------------- | --- | --- | ---- | --- | ----------- | --- | --- | --- | --- | --- | --- | --- |
noise.
| | | | | | | | F ExtendedDepthPilot | | | | | |
| --- | --- | --- | --- | --- | --- | --- | -------------------- | --- | --- | --- | --- | --- |
D.2 PairedBootstrapSignificanceTests We extended evaluation to depths 20, 30, and 50
onastratified40-scenariosubset(8percategory)
| We use paired | | bootstrap | tests | (10,000 | resamples) | | | | | | | |
| ------------- | --- | --------- | ----- | ------- | ---------- | --- | --- | --- | --- | --- | --- | --- |
forthreemodelsexhibitingthewidestperformance
| to assess | whether | differences | | between | | adjacent | | | | | | |
| --------- | ------- | ----------- | --- | ------- | --- | -------- | --- | --- | --- | --- | --- | --- |
spread.
| models in | the ranking | | are statistically | | significant. | | | | | | | |
| -------------- | ----------- | ----------- | ----------------- | --- | ------------ | ------- | -------- | ------------------------------------- | --- | --- | --- | --- |
| The difference | | between | Gemini-3.1-Pro | | | and all | | | | | | |
| | | | | | | | Table12: | Retention(%)atextendeddialoguedepths. | | | | |
| other models | is | significant | at | p < | 0.001. | Differ- | | | | | | |
encesamongthetop-5models(DeepSeek-v4-Flash Model Turn12 Turn20 Turn30 Turn50
throughGPT-5.4)arenotindividuallysignificantat
| | | | | | | | DeepSeek-v4-Flash | | 100.0 | 100.0 | 97.5 | 95.0 |
| --- | --- | --- | --- | --- | --- | --- | ----------------- | --- | ----- | ----- | ---- | ---- |
p < 0.05,consistentwiththesemodelsoccupying GPT-5.4 96.0 92.5 87.5 82.5
| | | | | | | | Gemini-3.1-Pro | | 54.0 | 47.5 | 40.0 | 32.5 |
| --- | --- | --- | --- | --- | --- | --- | -------------- | --- | ---- | ---- | ---- | ---- |
asharedhigh-retentiontier.
| | | | | | | | Table14: | Retention(%)withSSAvs.baselineatdepth | | | | | |
| ----------------------------------------- | ---------------- | ----- | -------- | ------- | --- | -------- | -------- | ------------------------------------- | -------- | ---------- | ---- | -------- | --- |
| Within | this 40-scenario | | pilot, | even | the | most | | | | | | | |
| stablesampledmodelshowsmeasurabledegrada- | | | | | | | 12. | | | | | | |
| tion at depth | 50 | (−5.0 | points). | GPT-5.4 | | exhibits | | | | | | | |
| | | | | | | | Model | | Baseline | +Reinstate | +SSA | ∆vs.Base | |
roughlylineardecayof∼0.35pointsperturnbe-
yonddepth12,whileGemini-3.1-Prolosesanad- GPT-5.4 96.0 97.6 99.2 +3.2
| | | | | | | | Claude-Sonnet-4-6 | | 88.0 | 92.0 | 96.0 | +8.0 | |
| --- | --- | --- | --- | --- | --- | --- | ----------------- | --- | ---- | ---- | ---- | ---- | --- |
ditional21.5pointsfromdepth12to50. Integrity Gemini-3.1-Pro 54.0 65.6 78.4 +24.4
| | | | | | | | Qwen3.6-Plus | | 96.0 | 97.6 | 98.4 | +2.4 | |
| --- | --- | --- | --- | --- | --- | --- | ------------ | --- | ---- | ---- | ---- | ---- | --- |
failuresconstitute91.3%ofcollapsesatdepth30–
| 50 (vs. 88.6% | at | depth | ≤12), | suggesting | | that the | | | | | | | |
| ------------- | --- | ----- | ----- | ---------- | --- | -------- | --- | --- | --- | --- | --- | --- | --- |
recall–compliancedissociationcanwidenatlonger
| | | | | | | | and 0.2 | on GPT-5.4. | | In dynamic | dialogues, | | how- |
| --- | --- | --- | --- | --- | --- | --- | ------- | ----------- | --- | ---------- | ---------- | --- | ---- |
horizons.
ever,blindre-anchoringcanover-enforcestalecon-
straints;adeploymentsystemmustfirstupdatethe
G Cross-LingualPilot
active-stateledgerwhenauserrevokesorrevisesa
| Wetranslated30representativescenarios(6percat- | | | | | | | rule. | | | | | | |
| ---------------------------------------------- | --- | --- | --- | --- | --- | --- | ----- | --- | --- | --- | --- | --- | --- |
egory)intoMandarinChineseandJapanese,with
| | | | | | | | I AdditionalMetricandDynamic-State | | | | | | |
| --------------- | --- | ------------ | --- | ----------- | --- | -------- | ---------------------------------- | --- | --- | --- | --- | --- | --- |
| native-speaker | | verification | of | translation | | fidelity | | | | | | | |
| andnaturalness. | | | | | | | Audits | | | | | | |
Tostress-testfivemethodologicalconcerns,weask
| Table13: | Cross-lingualretention(%)ona30-scenario | | | | | | | | | | | | |
| -------- | --------------------------------------- | --- | --- | --- | --- | --- | ---------------------------------------------- | ---- | ---------------- | --- | -------- | ------- | --- |
| subset. | | | | | | | whetherstrictjointretentionconflatesimmediate- | | | | | | |
| | | | | | | | task failure | with | persistent-state | | failure, | whether | |
Model English Chinese Japanese signed SIC components can be reported more di-
GPT-5.4 93.3 90.0 86.7 rectly by category, whether the symbolic-depth
| Claude-Sonnet-4-6 | | 90.0 | | 86.7 | 83.3 | | | | | | | | |
| ----------------- | -------- | -------- | --- | -------- | ------ | --- | --------------- | ------- | ---------- | ------------- | ------- | ------- | ---- |
| | | | | | | | signal persists | | when | the brittle | Gemini | outlier | is |
| Gemini-3.1-Pro | | 63.3 | | 56.7 | 53.3 | | | | | | | | |
| | | | | | | | excluded, | whether | SSA | over-enforces | | stale | con- |
| | | | | | | | straints | after | revocation | or | update, | and how | the |
| The pilot | observes | SIC-like | | failures | across | all | | | | | | | |
diagnosischangeswhentwopersistentconstraints
| three languages | | with preserved | | model | rankings | | | | | | | | |
| --------------- | --- | -------------- | --- | ----- | -------- | --- | ------------------------ | --- | --- | ------------------ | --- | --- | --- |
| | | | | | | | areactivesimultaneously. | | | Wethereforerancom- | | | |
(τ = 0.87). Retention drops 3–7 points from pactfollow-upauditsusingthesameloggedmodel
| English to | Chinese | and | an additional | | 3–4 | points | | | | | | | |
| ---------- | ------- | --- | ------------- | --- | --- | ------ | --- | --- | --- | --- | --- | --- | --- |
snapshots,withoneauditintentionallyexcluding
| to Japanese. | Japanese | | showed | distinctive | | per- | | | | | | | |
| ------------ | -------- | --- | ------ | ----------- | --- | ---- | --- | --- | --- | --- | --- | --- | --- |
Gemini,andsavedallrawoutputs,judgedecisions,
sona degradation (−8.9 points vs. English) due andsummarieswiththereleasedexperimentlogs.
| to competing | politeness | | norms. | Chinese | | showed | | | | | | | |
| ------------ | ---------- | --- | ------ | ------- | --- | ------ | --- | --- | --- | --- | --- | --- | --- |
the largest numerical constraint weakness (−5.6 I.1 StrictJointversusConstraint-Only
| points), potentially | | from | numeral-classifier | | | inter- | Scoring | | | | | | |
| -------------------- | --- | ---- | ------------------ | --- | --- | ------ | ------- | --- | --- | --- | --- | --- | --- |
actions. Becausethesubsethasonly30scenarios
Table15reportsa70-outputaudit(10probesper
| and 3 models, | these | are | diagnostic | | signals | rather | | | | | | | |
| ------------- | ----- | --- | ---------- | --- | ------- | ------ | --- | --- | --- | --- | --- | --- | --- |
model)thatseparatelyscoresimmediate-tasksuc-
thandefinitivemultilingualestimates.
| | | | | | | | cess, constraint-only | | | compliance, | strict | joint | suc- |
| --- | --- | --- | --- | --- | --- | --- | --------------------- | --- | --- | ----------- | ------ | ----- | ---- |
cess,recall,SIC+,andSIC−.
Theauditverifiesthe
H StructuredStateAnchoring(SSA)
| | | | | | | | interpretationusedinthemaintext: | | | | | strictjointre- | |
| --- | --- | --- | --- | --- | --- | --- | -------------------------------- | --- | --- | --- | --- | -------------- | --- |
SSA operates in three stages: (1) structured tentionisaconservativedeploymentmetric,while
[ACTIVE_STATE]prependingateachturn;(2)con- constraint-onlycomplianceisolatesthepersistent-
straint reformulation as verifiable predicates; (3) rulecomponent. Forexample,severalmodelspre-
post-generation verification with up to 2 retries. serve the constraint more often than they satisfy
Thisstresstestassumesthattheoriginalconstraint thefulljointtask,sotreatingstrictjointretention
remainsactivethroughoutthedialogue. Ittherefore aspurememorywouldbeoverlypessimistic.
| measures | recovery | under | immutable | | constraints, | | | | | | | | |
| -------- | -------- | ----- | --------- | --- | ------------ | --- | --------------------------------- | --- | --- | --- | --- | --- | --- |
| | | | | | | | I.2 SignedSICComponentsbyCategory | | | | | | |
notfalse-positiveover-enforcementafteruserrevo-
| cationorupdate. | | | | | | | Table16reportsthesignedSICcomponentsbycat- | | | | | | |
| --------------- | --- | --- | --- | --- | --- | --- | ------------------------------------------ | --- | --- | --- | --- | --- | --- |
SSArecoverssubstantialcompliancewithoutre- egoryonthesamecompactforkedaudit. Because
trainingwhentheactiveconstraintisstillvalid. The thisauditcontainsonly14probespercategory,we
largestgainisonGemini-3.1-Pro(+24.4points). use it as a diagnostic check rather than a replace-
SSAaddsmean1.3retriesperviolationonGemini mentfortheprimaryfailure-sidedecompositionin
Table15: Compactstrict-jointversusconstraint-onlyauditoncontrolledmulti-turnhistories(10probes/model).
Immediate-tasksuccessisjudgedwhileignoringthepersistentconstraint;constraint-onlyusesthescenariochecker;
strictjointrequiresboth.
| | | | | | | | | | SIC+ | SIC− |
| ----------------- | --- | --- | --- | ---- | ---------- | ------ | ----- | ------ | ------- | ---- |
| Model | | | n | Task | Constraint | Strict | joint | Recall | | |
| GPT-5.4 | | | 10 | 80.0 | | 100.0 | 80.0 | 100.0 | 0.0 | 0.0 |
| GPT-5.4-mini | | | 10 | 80.0 | | 90.0 | 70.0 | 100.0 | 10.0 | 0.0 |
| Claude-Sonnet-4-6 | | | 10 | 80.0 | | 60.0 | 40.0 | 10.0 | 10.0 | 60.0 |
| Gemini-3.1-Pro | | | 10 | 40.0 | | 30.0 | 20.0 | | 0.0 0.0 | 30.0 |
DeepSeek-v4-Pro
| | | | 10 | 60.0 | | 90.0 | 50.0 | 90.0 | 10.0 | 10.0 |
| ----------------- | --- | --- | --- | ---- | --- | ----- | ---- | ----- | ---- | ---- |
| DeepSeek-v4-Flash | | | 10 | 50.0 | | 100.0 | 50.0 | 100.0 | 0.0 | 0.0 |
| Qwen3.6-Plus | | | 10 | 90.0 | | 100.0 | 90.0 | 100.0 | 0.0 | 0.0 |
Table16: SICcomponentsbyconstraintcategoryinthe Table17: Gemini-excludedsymbolic-depthdiagnostic
compactforkedaudit. Valuesarepercentagesover14 audit. C@d is deterministic constraint compliance at
| probespercategory(2scenarios×7models). | | | | | | depthd;∆isC@50–C@12. | | | | |
| -------------------------------------- | --- | --- | --- | --- | --- | -------------------- | --- | --- | --- | --- |
Category n Constraint Recall SIC+ SIC− Model C@12 C@30 C@50 ∆ SIC+@50
Numerical 14 71.4 71.4 7.1 7.1 GPT-5.4 91.7 100.0 91.7 0.0 8.3
Format 14 78.6 78.6 7.1 7.1 GPT-5.4-mini 91.7 83.3 91.7 0.0 8.3
Persona 14 71.4 71.4 7.1 7.1 Claude-Sonnet-4-6 91.7 100.0 91.7 0.0 8.3
| | | | | | | DeepSeek-v4-Pro | | 100.0 91.7 | 83.3 -16.7 | 16.7 |
| ------- | --- | --- | --------- | --- | ---- | ----------------- | --- | ----------- | ---------- | ---- |
| Factual | 14 | | 92.9 64.3 | 0.0 | 28.6 | | | | | |
| | | | | | | DeepSeek-v4-Flash | | 91.7 100.0 | 83.3 -8.4 | 16.7 |
| Logical | 14 | | 92.9 71.4 | 0.0 | 21.4 | | | | | |
| | | | | | | Qwen3.6-Plus | | 100.0 100.0 | 100.0 0.0 | 0.0 |
| | | | | | | Aggregate | | 94.4 95.8 | 90.3 -4.1 | 9.7 |
Table6. Itneverthelessmakesthesignedquantities
| explicit: | SIC+ | captures | recalled-but-violated | | be- | | | | | |
| --------- | ---- | -------- | --------------------- | --- | --- | --- | --- | --- | --- | --- |
havior,whereasSIC−
capturesbehaviorallycom- ports a calibrated conclusion: the strongest fail-
pliantoutputswhoserecallprobefails.
uresaremodel-dependent,butthesymbolicrecall–
compliancedissociationisnotexclusivelyaGem-
I.3 Gemini-ExcludedSymbolic-DepthCheck
iniartifact.
| To further | isolate | | the model-dependency | | con- | | | | | |
| ---------- | ------- | --- | -------------------- | --- | ---- | --- | --- | --- | --- | --- |
cern,werananewdiagnosticauditthatexcludes
Gemini-3.1-Pro entirely and focuses on sym- I.4 DynamicUpdateandRevocationStress
| bolic constraints | | at | longer depths. | The | audit | Test | | | | |
| ----------------- | -------------- | --- | -------------- | ------ | ------- | ---- | --- | --- | --- | --- |
| covers | six non-Gemini | | models, | twelve | numeri- | | | | | |
cal/format/logicalscenarios,threedepths(12,30, Table 18 reports a 7-model × 5-scenario × 3-
and50),andforkedbehavior/recallbranches,yield- condition stress test for dynamic-state handling.
ing 216 paired probes (432 real model calls). Ta- The base condition gives the model the dialogue
ble17showsthataggregateconstraintcompliance historyaftertheuserexplicitlyrevokesorupdates
remains high but not perfect: 94.4% at depth 12, a constraint. The stale-SSA condition intention-
95.8% at depth 30, and 90.3% at depth 50. The ally injects the old active-state ledger, and the
depth-50 SIC+ rate is 9.7%, and five of the six updated-ledgerconditioninjectstherevisedledger.
non-Gemini models exhibit at least one recalled- The base dialogue rarely enforces the stale rule
but-violatedcase. Adescriptivemechanismaudit (2.9%),butstaleSSAraisesstaleenforcementto
furtherlocalizesthegap: numericalconstraintsre- 74.3%. Updating the active-state ledger before
tain100.0%recallbutonly79.2%complianceat re-anchoring reduces stale enforcement to 0.0%
depth50,yieldingSIC+ = 20.8%;allfivenumeri- and yields updated/clean behavior in 97.1% of
calSIC+
eventsoccurinresponse-internalcount- cases. This is why our SSA claim is scoped to
ing cases. This pattern is consistent with an on- immutableactiveconstraintsandwhydeployment
lineexecutionorself-verificationbottleneckrather systemsneedexplicitstate-updatetrackingbefore
| thansimplelossoftherulerepresentation. | | | | | Itsup- | re-anchoring. | | | | |
| -------------------------------------- | --- | --- | --- | --- | ------ | ------------- | --- | --- | --- | --- |
| Table18: | Dynamicrevoke/updatestresstestforSSA. | | | | | | | | | | | | |
| -------- | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
SSA,aftertheuserupdatedthesuffixrulefromEND
Staleenforcedcountsoutputsthatincorrectlyfollowthe toDONE,staleSSAproducedananswerendingin
| old constraint | after | it | was revoked | or | updated. | Base | | | | | | | |
| -------------- | ----- | --- | ----------- | --- | -------- | ---- | --- | --- | --- | --- | --- | --- | --- |
END,whileupdated-ledgerSSAproducedthesame
usesonlythedialoguehistory;staleSSAintentionally
answerendinginDONE.Thesecasesareillustrative;
injectstheoldactivestate;updated-ledgerSSAinjects
thequantitativelabelscomefromthefullcompact
therevisedactivestate.
auditabove.
| Condition | | n Staleenforced | | Stale% | Updated/clean% | | | | | | | | |
| ----------------- | --- | --------------- | --- | ------ | -------------- | ---- | ----------------------------- | -------- | ----- | ------------ | --- | ----- | --- |
| | | | | | | | J Multi-JudgeCross-Validation | | | | | | |
| Baseafterchange | | 35 | 1 | 2.9 | | 94.3 | | | | | | | |
| StaleSSAledger | | 35 | 26 | 74.3 | | 20.0 | | | | | | | |
| Updated-ledgerSSA | | 35 | 0 | 0.0 | | 97.1 | | | | | | | |
| | | | | | | | To address | concerns | about | single-judge | | bias, | we |
re-scoredastratified200-outputsubsetusingthree
| Table 19: | Compact | two-constraint | | diagnostic | | audit. | | | | | | | |
| --------- | ------- | -------------- | --- | ---------- | --- | ------ | ------------------------------ | --- | --- | --- | ----------------- | --- | --- |
| | | | | | | | architecturallydistinctjudges: | | | | GPT-5.4(primary), | | |
C1/C2arethetwodeterministicconstraintchecks;Joint
requires both constraints; Recall is the forked recall Claude-Sonnet-4-6,andGemini-3.1-Pro,eachwith
thesamefixedevaluationpromptattemperature0.
probe.
| | | | | | | | Table 20: | Multi-judge | agreement | | and self-preference | | |
| ----- | --- | --- | --- | -------- | ------ | ---- | --------- | ----------- | --------- | --- | ------------------- | --- | --- |
| Model | | n | C1 | C2 Joint | Recall | SIC+ | | | | | | | |
analysison200-outputsubset.
| GPT-5.4 | | 4 | 100.0 100.0 | 100.0 | 100.0 | 0.0 | | | | | | | |
| ----------------- | --- | --- | ----------- | ----- | ----- | --- | ----- | ------------------ | --- | ----------- | ----------- | -------- | ----- |
| GPT-5.4-mini | | 5 | 100.0 100.0 | 100.0 | 100.0 | 0.0 | | | | | | | |
| | | | | | | | Judge | Agreementw/Primary | | κ(pairwise) | Self-Pref.∆ | McNemarp | Bias? |
| Claude-Sonnet-4-6 | | 5 | 80.0 100.0 | 80.0 | 60.0 | 0.0 | | | | | | | |
Gemini-3.1-Pro 5 20.0 0.0 0.0 0.0 0.0 GPT-5.4(primary) — — +1.2% — —
DeepSeek-v4-Pro 5 100.0 80.0 80.0 60.0 20.0 Claude-Sonnet-4-6 94.0% 0.87 +0.8% 0.42 No
| | | | | | | | Gemini-3.1-Pro | 91.5% | | 0.82 | −0.4% | 0.68 | No |
| ----------------- | --- | --- | ----------- | ----- | ----- | ---- | ----------------- | ----- | --- | ---- | ----- | ---- | --- |
| DeepSeek-v4-Flash | | 5 | 80.0 100.0 | 80.0 | 100.0 | 20.0 | | | | | | | |
| | | | | | | | Fleiss’κ(3judges) | | | 0.89 | | | |
| Qwen3.6-Plus | | 5 | 100.0 100.0 | 100.0 | 100.0 | 0.0 | | | | | | | |
Aggregate 34 82.4 82.4 76.5 73.5 5.9 Self-preference ∆ measures whether a judge
| | | | | | | | scores | outputs from | its | own model | | family | more |
| --- | --- | --- | --- | --- | --- | --- | --------- | ------------ | --- | ---------- | ------- | ------ | ---- |
| | | | | | | | leniently | than outputs | | from other | models. | | The |
I.5 Two-ConstraintScopeCheck
| | | | | | | | maximum | observed | ∆ | is +1.2% | (GPT-5.4 | | judg- |
| --- | --- | --- | --- | --- | --- | --- | ------- | -------- | --- | -------- | -------- | --- | ----- |
Finally, Table 19 reports a low-cost diagnostic ingGPT-5.4outputs),whichisnotstatisticallysig-
micro-auditwithtwosimultaneouspersistentcon- nificant (McNemar’s p = 0.31). Model rankings
| straints | per dialogue. | | This | audit directly | | probes | | | | | | | |
| -------- | ------------- | --- | ---- | -------------- | --- | ------ | ------------- | ------ | --- | ------------- | --- | ------ | ---- |
| | | | | | | | are identical | across | all | three judges. | | The 6% | dis- |
the ecological-validity boundary of the single- agreementcasesconcentrateinrole/personaitems
constraintbenchmarkdesign. Across34validout- withsubtlestylisticdrift,whereClaude-Sonnet-4-
| puts (one | GPT-5.4 | probe | encountered | | a | repeated | | | | | | | |
| --------- | ------- | ----- | ----------- | --- | --- | -------- | --- | --- | --- | --- | --- | --- | --- |
6ismarginallymorelenientandGemini-3.1-Pro
| upstream | 502 | and is | excluded), | | the joint | two- | | | | | | | |
| -------- | --- | ------ | ---------- | --- | --------- | ---- | ---------- | ---------------- | --- | ------- | --- | ---------- | --- |
| | | | | | | | marginally | stricter—neither | | pattern | is | systematic | |
constraint success rate is 76.5%, lower than the enoughtoalteranymodel’stierplacement.
| main single-constraint | | | benchmark, | | while | forked | | | | | | | |
| ---------------------- | --- | --- | ---------- | --- | ----- | ------ | --- | --- | --- | --- | --- | --- | --- |
recallis73.5%. Thisconfirmsthatmulti-constraint K SSAInferenceCostAnalysis
| interference | is | a harder | setting | and | should | be | | | | | | | |
| ------------ | --- | -------- | ------- | --- | ------ | --- | ----------- | --- | ------- | --- | ----- | -------- | --- |
| | | | | | | | We quantify | the | latency | and | token | overhead | of |
treatedasfuturefull-benchmarkwork,whilepre-
StructuredStateAnchoringtoenableinformedde-
| servingthepaper’smaindesignrationale: | | | | | | thecore | | | | | | | |
| ------------------------------------- | --- | --- | --- | --- | --- | ------- | --- | --- | --- | --- | --- | --- | --- |
ploymentdecisions.
benchmarkintentionallyisolatesstateintegritybe-
foreaddinginteractingconstraints.
| | | | | | | | Table21: | SSAinferenceoverheadatdepth12. | | | | | Latency |
| --- | --- | --- | --- | --- | --- | --- | ------------- | ------------------------------ | ------------- | --- | ----- | ------------- | ------- |
| | | | | | | | is wall-clock | time | per response; | | token | cost includes | |
I.6 IllustrativeCases
input+outputtokensforallretries.
| The following | | compact | examples | | show | how the | | | | | | | |
| ------------- | --- | ------- | -------- | --- | ---- | ------- | --- | --- | --- | --- | --- | --- | --- |
auditlabelsareassigned. InaSIC+ case,thesetup Model BaseLat. SSALat. Lat.× BaseTok. SSATok. Tok.×
| | | | | | | | GPT-5.4 | 2.1s | 2.8s | 1.34× | 1,842 | 2,174 | 1.18× |
| --- | --- | --- | --- | --- | --- | --- | ------- | ---- | ---- | ----- | ----- | ----- | ----- |
requiredeveryresponsetoendwithanexactword Claude-Sonnet-4-6 2.4s 3.6s 1.50× 1,956 2,487 1.27×
| | | | | | | | Gemini-3.1-Pro | 1.8s | 3.8s | 2.12× | 1,724 | 2,534 | 1.47× |
| --- | --- | --- | --- | --- | --- | --- | -------------- | ---- | ---- | ----- | ----- | ----- | ----- |
count in brackets. GPT-5.4-mini answered the Qwen3.6-Plus 1.9s 2.7s 1.42× 1,801 2,198 1.22×
weatherprobewithareasonablecaveatbutomitted
thebracketedcount,whileitsseparaterecallbranch Theoverheadismodel-dependentandcorrelates
SIC−
correctly restated the rule. In a case under withbaselineSICseverity: modelsthatfailmore
the same setup, Claude-Sonnet-4-6 produced a often trigger more retries. GPT-5.4 averages 0.2
bracketed-count answer, but its recall branch re- retries per response (mean 1 retry every 5 turns),
fused to state the active constraint. For dynamic while Gemini-3.1-Pro averages 1.3 retries. The
structured prepending component alone (no veri- experiment summary files used to produce
fication/retry)addsonly12–15%tokenoverhead the tables. From the release root, python
andrecovers60–70%oflostcompliance,makingit tools/reproduce_reported_metrics.py re-
suitableforlatency-sensitivedeployments. Thefull constructs the primary seven-model retention
SSApipeline(prepending+verification+retry)is table and the dynamic-state, symbolic-depth,
recommendedforhigh-stakesapplicationswhere execution-bottleneck, and two-constraint results
correctnessoutweighslatency. from cached JSON files; python -m unittest
| | | | | | | | discover | -s | tests | -v runs | the credential-free | |
| ----------------------- | ----------- | --- | ------------ | ---------------- | -------------- | --- | ---------- | --------------------- | ----- | ------- | ------------------- | ---- |
| Componentdecomposition. | | | | SSA’sthreestages | | | | | | | | |
| | | | | | | | evidence | checks. | For | closed | API systems, | the |
| contribute | differently | | to recovery: | | (1) structured | | | | | | | |
| | | | | | | | logs store | the endpoint/snapshot | | | identifiers | from |
prependingalonerecovers62%oflostcompliance
Table1sothatfuturere-executionscandistinguish
onaverage(cost: +13%tokens,+8%latency);(2) provider drift from benchmark variation. All
addingconstraintreformulationasverifiablepred-
| | | | | | | | user-turn | content | was | authored | originally | by our |
| --------------- | --- | ------------- | --- | --- | ------ | ------- | --------- | -------- | --- | -------- | ---------- | ----------- |
| icates recovers | | an additional | | 14% | (cost: | +4% to- | | | | | | |
| | | | | | | | team; no | material | was | derived | from | proprietary |
kens,negligiblelatency);(3)post-generationver-
| | | | | | | | or copyrighted | | external | corpora. | Model | outputs |
| --------- | ---------- | ---------- | --- | --------- | --------- | ------ | -------------- | ------- | -------- | -------- | ---------- | -------- |
| ification | with retry | recovers | | the final | 24% | (cost: | | | | | | |
| | | | | | | | included | in case | studies | are | used under | fair-use |
| variable, | 0–47% | additional | | tokens | depending | on | | | | | | |
provisionsfornon-commercialacademicresearch.
| retryfrequency). | | Thisdecompositiondemonstrates | | | | | | | | | | |
| ---------------- | --- | ----------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
Crowdworkannotatorswerecompensatedat$18
| that SSA | is not | merely | “retry | until | correct” | but a | | | | | | |
| --------------------------- | ------ | ------ | ---------------------- | -------------------- | -------- | ----- | ------------- | --- | ----------------------------- | --- | ---- | ------------ |
| | | | | | | | USDperhour. | | Allscenariocontentwasreviewed | | | |
| principleddiagnostictool: | | | theprependingstageiso- | | | | | | | | | |
| | | | | | | | independently | | for demographic | | bias | and cultural |
| latesstate-bindingfailures, | | | | whiletheverification | | | | | | | | |
stereotypingbeforeinclusion.
| stage isolates | generation-time | | | constraint | | enforce- | | | | | | |
| -------------- | --------------- | --- | --- | ---------- | --- | -------- | --- | --- | --- | --- | --- | --- |
mentfailures.
L PromptSensitivityAnalysis
Toverifythatbenchmarkconclusionsarerobustto
| prompt | formulation, | | we created | three | alternative | | | | | | | |
| --------- | ------------ | --- | ----------- | ----------- | ----------- | ----- | --- | --- | --- | --- | --- | --- |
| phrasings | for each | of | 50 randomly | | sampled | con- | | | | | | |
| straints | (maintaining | | semantic | equivalence | | while | | | | | | |
varyingsurfaceform,specificitylevel,andimpera-
tivevs.declarativeframing).
| Table22: | Retention(%)underalternativepromptfor- | | | | | | | | | | | |
| -------- | -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
mulations(50-scenariosubset).
| Model | | Original | VariantA | VariantB | | VariantC | | | | | | |
| ----------------- | --------- | -------- | -------- | -------- | -------- | -------- | --- | --- | --- | --- | --- | --- |
| GPT-5.4 | | 94.0 | 93.2 | | 94.8 | 92.4 | | | | | | |
| Claude-Sonnet-4-6 | | 90.0 | 88.8 | | 91.2 | 89.6 | | | | | | |
| Gemini-3.1-Pro | | 62.0 | 60.4 | | 63.6 | 59.2 | | | | | | |
| Maximum | deviation | | across | prompt | variants | is | | | | | | |
3.6points(Gemini,VariantC),andmodelrankings
| are preserved | across | | all formulations | | (τ | = 1.0). | | | | | | |
| ------------- | ------ | --- | ---------------- | ------- | ----- | ------- | --- | --- | --- | --- | --- | --- |
| This confirms | that | SIC | findings | reflect | model | be- | | | | | | |
haviorratherthanprompt-specificartifacts.
StateTrack-Benchscenariosandderivedbench-
| mark metadata | | are released | | under | CC | BY 4.0, | | | | | | |
| ------------- | --- | ------------ | --- | ----- | --- | ------- | --- | --- | --- | --- | --- | --- |
andtheevaluationcodeisreleasedundertheMIT
License. Thecompletereleasepackageisavailable
at: https://github.com/xzzvsxd/StateTrack.
| It includes | the | scenario | | definitions, | | exact | | | | | | |
| ----------- | -------- | ------------ | --- | ------------ | ---------- | ----- | --- | --- | --- | --- | --- | --- |
| model | prompts, | judge | | prompts, | evaluation | | | | | | | |
| scripts, | raw | model-output | | logs, | and | per- | | | | | | |