← Back to summary
Reva: Real-Time Vulnerability Detection through Contextual Probing of Large Language Models
Converted from PDF with markitdown. Layout may be rough.
R : Real-Time Vulnerability Detection through
EVA
Contextual Probing of Large Language Models
Amine Lbath, Massih-Reza Amini,
Université Grenoble Alpes, CNRS, France
Abstract—A vulnerability costs more to fix the later it is the function-level detection literature, and VulAgent [7] enrich
discoveredinthesoftwarelifecycle.Ascodingagentsnowgenerate thepromptwithretrievedvulnerabilityknowledgeandprogram
codefasterthandeveloperscanreviewit,softwareneedsastronger
facts, and VulTrial [8] lets several agents debate a verdict.
first line of defense that screens each new function in real time,
This reasoning, however, spans several model calls, which
as it is generated. LLM agents have recently become capable of
finding real-world vulnerabilities by working as a security expert amounts to minutes per function. Faster AI-based detectors,
would: they run analysis tools, explore the surrounding code, which fine-tune deep learning models or prompt an LLM for
test hypotheses and weigh the evidence before forming their own a direct verdict, reduce this latency but are considerably less
judgement. This reasoning, however, spans many model calls,
accurate [9]–[11].
so that each verdict takes minutes of compute, far too long to
keeppacewithcodegeneration.FasterAI-baseddetectors,which
REVA is designed to close this gap between speed and
fine-tune deep learning models or prompt an LLM for a direct improved accuracy. Instead of letting the LLM gather context
verdict, are considerably less accurate: they judge a function through multiple rounds of tool calls, REVA assembles it in
without the context an agent gathers, and an LLM’s generated the prompt, including a threat model of the function’s module,
verdict has proven unreliable in prior work even when some of
its static analysis findings, its call graph and its file. Instead
that context is given.
of reasoning towards a verdict, the LLM then processes this
We present REVA (REal-time Vulnerability Analysis), an AI
vulnerability detector that brings the security context and code prompt in a single forward pass without generating text, and
understanding components behind agentic detectors into a single a small probe trained on labeled functions maps its internal
forward pass of an LLM. For each new function, REVA provides representations of the code tokens to a risk score. This second
the LLM with a pre-computed threat model of its module,
choice builds on evidence that LLMs encode knowledge that
along with lightweight static analysis findings and call-graph
their answers do not express, for example about the truth of a
information. Rather than asking the LLM for a verdict, REVA
lets a trained probe read the LLM’s internal representations of statement[12]–[14]orthecorrectnessoftheirownanswers[15],
thefunctioninthiscontext.OnPrimeVul’sheld-outtestset,REVA [16], and that chain-of-thought explanations are not always
on GLM-5.3-Flash reaches 60.0% accuracy in about one second faithful to the underlying computation [17].
perfunction.ThismatchesVulTriage(59.2%),thestate-of-the-art
We evaluate REVA on PrimeVul [9], a high-quality dataset
agentic function-level detector in the literature, run on the same
ofvulnerableCandC++functionspairedwithbenignversions
LLM, while being over 250 times faster and less than a tenth
as costly. On the same prompt, the probe is 14% to 17% more of the same functions, which spans 651 projects and 121
accurate than the LLM’s own direct verdict. Across three open vulnerability types (CWEs). On its held-out test set, REVA
LLMs, REVA is at least as accurate as VulTriage, and about on GLM-5.3-Flash [18] is on par with VulTriage and more
10% more accurate on two of them. These results suggest that
accurate than VulAgent run on the same LLM, in about one
an LLM’s internal representations already encode much of what
second per function, over 250 times faster than VulTriage. On
agentic detectors obtain through reasoning.
Index Terms—vulnerability detection, large language models, two other open LLMs, it is 9% to 13% more accurate than
internal representations, probing, coding agents both agentic detectors.
This paper makes three contributions: (i) we propose REVA,
I. INTRODUCTION
a vulnerability detector that combines targeted security context
The cost of remediating a vulnerability grows the later it with a probe on an LLM’s internal representations, in a single
is discovered in the software lifecycle [1]. Vulnerabilities are forward pass per function; (ii) we compare it with state-of-
therefore cheapest to fix when they are detected as the code is the-art agentic function-level detectors run on the same three
written.Codingagentsnowgeneratecodefasterthandevelopers LLMs, on PrimeVul’s held-out test set, in terms of accuracy,
canreviewit,whichcallsforafirstlineofdefensethatscreens cost and robustness; and (iii) we outline a research agenda
each new function in real time, as it is generated. towards detectors that screen every function as coding agents
Static analyzers run in real time, but at the scale of agent- generate it and escalate likely vulnerabilities to more capable
generated code, they produce more reports than developers agents that construct a proof of vulnerability.
can review, many of them false positives [2], [3]. LLM
agents have recently become capable of finding real-world
II. BACKGROUNDANDRELATEDWORK
vulnerabilities [4], [5] by combining such tools with their own
analysis of the code, and agentic detectors bring this approach LLM-based vulnerability detection. LLMs in non-agentic
to function-level detection. VulTriage [6], the state of the art in settings typically struggle to tell a vulnerable function from
6202
tcO
62
]RC.sc[
1v92853.0162:viXra
①Promptwithsecuritycontext ②Oneforwardpass ③Code-tokenstates ④Attentionprobe→riskscore
| ## Threat | model: project, | module | | not | ℓ4 | | | | |
| ---------- | ----------------------- | ------ | --- | -------- | --- | --- | --- | --- | --- |
| ## Semgrep | findings | | | executed | | | | | |
| ## Call | graph: callers, callees | | | | ℓ3 | | | | |
con-
| | | | | | | | cat | learnedattentionove | r c o d e t o k e n s |
| --- | --- | --- | --- | --- | --- | --- | --- | ------------------- | --------------------- |
## File: includes, types, macros ℓ2 f u n c ti o n ’ s r is kscore
## New function
| node *new_node(tree | *tr) { | | | probed | ℓ1 | | | | |
| ------------------- | -------------------- | --- | --- | ------ | --- | --- | --- | --------------- | --- |
| node *t | = &tr->nodes[tr->n]; | | | | | | | | |
| tr->n++; | return t; } | | | | | | | medianthreshold | |
securitycontextaroundthenewfunction LLM,notextgenerated Tcodetokens×4·1,024 probeof0.57Mparameters,trainedonlabels
OverviewofREVAonavulnerablefunctionfromLibRaw(CVE-2020-35530).①Thepromptcombinesthefunctionwithitssecuritycontext.②The
Fig.1.
LLMprocessesthepromptinoneforwardpass,uptothedeepestoffourprobedlayers.③Thehiddenstatesofthecodetokensarekeptateachprobedlayer.
④Anattentionprobepoolsthemintoariskscore.
its benign counterpart: on PrimeVul [9], GPT-4 with chain-of- findings for the function, its call sites and the signatures of
thought labels both versions correctly in only 12.9% of the the helpers it calls, and the includes, types, macros and sibling
pairs, and LLMs asked for a verdict in non-agentic settings, functionsofitsfile,extractedwithtree-sitterfromtherepository.
often reason inconsistently about security [19], [20]. REVA It is available for 84% of the training functions; otherwise the
instead reads the LLM’s internal representations rather than its prompt has the function and its Semgrep findings only. The
answer or reasoning. median prompt has 3,840 tokens, 601 of them code.
②
Agentic detectors. Recent detectors improve accuracy by One forward pass. The LLM processes the prompt in
adding knowledge and reasoning steps. VulTriage retrieves one forward pass that stops after the deepest of four probed
CWE descriptions and verbalizes control- and data-flow layers, so the remaining layers are never run and it generates
facts[6],VulAgentletsspecializedagentsproposeandvalidate no output token. We probe intermediate layers, which probing
hypotheses against the project’s code [7], and VulTrial stages studies often find most informative [21]: four evenly spaced
a debate between agents [8]. Their cost grows with each call, layers from 38% to 85% of the depth of the LLM, selected by
| andtheircontextcanexceedtheLLM’scontextwindow. | | | | REVA | cross-validation. | | | | |
| ---------------------------------------------- | --- | --- | --- | ---- | ----------------- | --- | --- | --- | --- |
③
instead pre-loads the key parts of this context into the prompt Code-token states. At each probed layer, REVA keeps
and replaces the multi-step reasoning with a single probed only the hidden states of the function’s tokens. Each state is
forward pass. normalized and projected to 1,024 dimensions, and the four
Probing LLM internal representations. Probes are small layers are concatenated.
classifiers trained on a model’s hidden states [21]. For code, ④Attentionprobe.Aprobewith8attentionheadspoolsthe
linear probes indicates some capability to separate secure from token states into one risk score. It has 0.57M parameters and
insecure code on toy datasets [22], [23], and predict whether is trained on PrimeVul’s training functions. Its loss combines
generated code will be insecure or incorrect [24], [25]. These a per-function classification term with a term that ranks each
probes read the code alone, whereas REVA probes real-world vulnerable function above its benign counterpart. These pairs
functions read with targeted security context. serve only in training: REVA scores each function on its own.
| | | | | | Use in | practice. | | | |
| --- | --- | --- | --- | --- | ------ | --------- | --- | --- | --- |
Pitfalls of vulnerability benchmarks.Detectorscanexploit The LLM and Semgrep’s rules are loaded
spurious features of their datasets, such as code metrics or once at startup, so they don’t have to be booted for each new
semantics unrelated to the vulnerability [26], [27]. Such a analysis, and each project’s threat model is built ahead of
detector tends to classify a vulnerable function and its benign time. For each new function, REVA extracts its context, runs
counterpart with the same label, which share these features. Semgrep, builds the prompt, runs the pass and applies the
| | | | | | probe. With | Muse-Glimmer-30B | | on one H200 | GPU, this takes |
| ------------ | ---------- | ----------------------- | --- | ---------- | ----------- | ---------------- | --- | ----------- | --------------- |
| We therefore | score each | function independently, | | then check | | | | | |
whether the two versions of each pair receive their correct 0.93s per function, 0.46s of which is context extraction.
labels.
IV. PRELIMINARYRESULTS
III. CONTEXTUALPROBINGWITHREVA We evaluate REVA through three research questions:
In REVA, an LLM reads a new function with its security RQ1: How accurate is REVA compared with agentic
| | | | | | detectors | running | on the | same LLM? | |
| ---------- | ----------------- | ----------- | ---- | ---------- | --------- | ------- | ------ | --------- | --- |
| context in | one forward pass, | and a probe | maps | the hidden | | | | | |
states of the code tokens to a risk score (Fig. 1). RQ2: How much computation does REVA save?
① Prompt with security context. The prompt contains the RQ3: Which parts of REVA matter, and is its accuracy
| | | | | | robust | to leakage | and selection? | | |
| ------------- | ------------------- | ---------- | ---------- | ----------- | --------------- | ---------- | -------------- | --- | --- |
| new function | and the context | around it, | in labeled | sections | | | | | |
| (Fig. 1). A | threat model, built | once per | project by | a workflow | | | | | |
| | | | | | A. Experimental | Setup | | | |
| of LLM agents | over the entire | software, | describes | the project | | | | | |
(summary, attacker-controlled inputs, trust boundaries) and the Dataset and metrics. PrimeVul [9] pairs each vulnerable
module that contains the function (role, inputs that reach it, C/C++ function with a benign version of the same function. It
typical sinks and guards). The prompt adds the Semgrep [28] has3,789training,480validationand435testpairs.Following
65
60
55
50
0.1 1 10 100 1,000 10,000
seconds per function (log scale)
)%(
ycarucca
Input
no repository context −1.0
function only −1.9
Decision
ask the LLM (Yes/No), no probe −9.4
Semgrep alone −11.0
Probe
Reva (one pass) single layer −0.4
VulTriage VulAgent
mean pooling −4.2
random BCE only (no ranking loss) −1.0
no standardization −1.5
−15 −10 −5 0 5
Accuracy difference from full Reva (points)
Fig. 2. Accuracy versus time per function on the PrimeVul test set, one
colorperLLM.Functionswithoutaverdictcountasbenign.Hollowmarker: Fig.3. ChangeinvalidationaccuracyofREVAonMuse-Glimmer-30Bwhen
VulTriageonGPT-4o,asreported. onecomponentisremovedorreplaced,withpaired95%confidenceintervals.
[9], we count the pairs in which both versions are labeled capturesthevulnerabilityitselfratherthansurfacefeaturesthat
correctly (P-C), both are flagged (P-V), both are cleared (P-B), a vulnerable function shares with its secured counterpart.
or the labels are reversed (P-R). Our main metric is accuracy Fig. 2 summarizes this trade-off. On each LLM, REVA is at
over all functions, each scored independently. least as accurate as both agentic detectors while being 100 to
Protocol. We made every design choice on the training and 6,000 times faster. Its accuracy also varies little across LLMs,
validationsetsonly.Cross-validation(CV)accuracycoversfive by1.1pointsagainst5.6forVulTriage,sothat REVA onMuse-
folds of the training set, with all functions of a commit in the Glimmer-30B matches VulTriage on GLM-5.3-Flash, the LLM
same fold, and the validation set, scored by a probe trained onwhichVulTriageperformsbest.Thesignalreadbytheprobe
on the full training set. The test set stayed unseen during is therefore present in all three LLMs, whereas the accuracy of
development and was scored only with the final configuration. the agentic detectors depends on how well each LLM reasons
LLMs. We use three open-weight LLMs: Muse-Glimmer- aboutthecode.VulTriageonGPT-4oreportsahigheraccuracy
30B[29],Qwen3.8-27B[30]andGLM-5.3-Flash[18],alarger (63.6%), but we were unable to replicate these results.
mixture-of-experts model. On GLM-5.3-Flash, the agentic
AnswertoRQ1:Fromasingleforwardpass, REVA matches
detectors use the FP8 version of the LLM and REVA an exact
or exceeds both function-level agentic detectors on all three
BF16 conversion, which leaves its cross-validation accuracy
LLMs. It is significantly more accurate in five of the six
unchanged (66.2% vs. 65.9%).
same-LLM comparisons and has fewer false positives than
Baselines.WerunVulAgentandVulTriageontheirreleased
VulTriage on every LLM.
code,usingthesameLLMsasREVA.Afunctionisfailedwhen
the detector returns no verdict, usually because the reasoning C. RQ2: Computational Cost
exceeds the token limit. We compare all methods over all
End to end, REVA takes 0.88s to 1.33s per function, against
functions, counting failed functions as benign. Published GPT-
108s to 5,614s for the agentic detectors on the same LLMs.
4o results are given for context.
Although these times, measured under parallel load, overstate
B. RQ1: Accuracy single-request latency, the token counts confirm the gap. REVA
reads about 3,100 to 3,400 tokens per function and generates
On GLM-5.3-Flash, REVA reaches 60.0% accuracy (95%
none,whereastheagenticdetectorsprocess3to70timesmore
CI [57.8, 62.2]), against 59.2% for VulTriage and 52.8%
for VulAgent run on the same LLM (Table I). Over all test
tokens, many of them generated. As REVA needs the LLM’s
hidden states, it cannot run behind an API. On a rented H200
functions, REVA is 0.8 points above VulTriage (CI [−2.1,
GPU, it screens 1,000 functions for $0.9 to $1.3, against $9 to
+3.8]) and 7.2 points above VulAgent (CI [+4.4, +10.3]). On
$61 for VulTriage and $63 to $307 for VulAgent at the public
Qwen3.8-27B and Muse-Glimmer-30B, REVA is 5.1 to 6.7
API rates of the same LLMs.
points more accurate than both agentic detectors, and all four
confidence intervals exclude zero. Answer to RQ2: REVA screens a function in about one
Beyond accuracy, REVA also compares favorably with second, against minutes for the agentic detectors, at 10 to
VulTriage, the state-of-the-art function-level agentic detector, 360 times lower cost on the same LLM, and returns a score
when both run on the same LLM. On all three LLMs, REVA for every function.
has a lower false positive rate (FPR) and inverts the labels of
D. RQ3: Ablation and Robustness
fewer pairs (P-R), and on two of them it labels more pairs
entirelycorrectly(P-C).REVAthusnotonlymatchesorexceeds We ablate each component of REVA on the Muse-Glimmer-
VulTriage in accuracy, but also separates vulnerable from 30Bvalidationset(Fig.3).AskingthesameLLMforaYes/No
benign functions more reliably, which suggests that the probe verdict on the same prompt, instead of probing it, reduces
TABLEI
ACCURACYANDCOSTOFREVAANDAGENTICDETECTORSONTHEPRIMEVULTESTSET.TOKENS:MEANINPUTANDOUTPUTTOKENSPERFUNCTION,IN
THOUSANDS.TIME:MEDIANSECONDSPERFUNCTION.CI:PAIR-BOOTSTRAPINTERVALONACCURACY.COST:USDPER1,000FUNCTIONS,FORREVAON
ONERENTEDH200GPUAT$3.50PERHOUR,ANDFORTHEAGENTICDETECTORSATPUBLICAPIRATESPERMILLIONINPUT/OUTPUTTOKENS
(MUSE-GLIMMER-30B$0.35/$1.50,QWEN3.8-27B$0.40/$3.00,GLM-5.3-FLASH$0.15/$0.50).FPR:SHAREOFNON-VULNERABLEFUNCTIONS
FLAGGED.–:NOTREPORTED.
MethodandLLM Calls Tokens(k) Time(s) Cost($) FPR Accuracy 95%CI P-C P-R
REVA(ours)
Muse-Glimmer-30B 1 3.2/0 0.93 0.9 41.1 58.9 [56.8,61.0] 20.2 2.5
| | Qwen3.8-27B | | | | | 1 | 3.4/0 | 0.88 | 0.9 | 40.5 59.5 | [57.4,61.7] | | 22.1 | 3.0 | |
| --- | ----------- | --- | --- | --- | --- | --- | ----- | ---- | --- | --------- | ----------- | --- | ---- | --- | --- |
GLM-5.3-Flash 1 3.1/0 1.33 1.3 40.0 60.0 [57.8,62.2] 22.3 2.3
Muse-Glimmer-30B+GLM-5.3-Flash 2 6.3/0 1.78 1.7 39.3 60.7 [58.5,62.9] 23.2 1.8
VulTriage[6]
Muse-Glimmer-30B 3 6.2/4.8 108 9.4 75.6 53.6 [51.5,55.7] 14.0 6.9
Qwen3.8-27B 3 8.0/19.4 564 61.4 75.4 53.9 [51.6,56.2] 16.1 8.3
GLM-5.3-Flash 3 6.2/24.7 362 13.3 46.0 59.2 [56.7,61.6] 26.0 7.6
GPT-4o(published,notreproducible) 3 – 34.5 – 54.3 63.6 – 33.8 6.7
VulAgent[7]
Muse-Glimmer-30B 9.2 34.1/33.8 785 62.6 64.1 52.2 [50.0,54.4] 13.3 9.0
Qwen3.8-27B 9.2 37.4/97.3 5,614 306.9 32.2 54.5 [52.0,57.0] 20.2 11.3
GLM-5.3-Flash 9.3 35.8/184.8 3,495 97.8 19.8 52.8 [50.5,54.9] 14.3 8.7
| | GPT-4o(published) | | | | | 8.8 | –/7.0 | – | – | 36.7 58.4 | | – | 26.6 | 10.1 | |
| --- | ----------------- | --- | --- | --- | --- | --- | ----- | --- | --- | --------- | --- | --- | ---- | ---- | --- |
VulTrial[8]
| | GPT-4o(published) | | | | | – | – | – | – | 52.6 53.6 | | – | 18.6 | 11.5 | |
| --- | ----------------- | --- | --- | --- | --- | --- | --- | --- | --- | --------- | --- | --- | ---- | ---- | --- |
accuracy to chance. On the test set, it stays 7.4 to 8.7 points V. FUTUREPLANS
belowtheprobe,andSemgrep,CppcheckandFlawfinderalone Step1:Completethecomparison.Wewillrunbothagentic
| are also | at | chance (50.8% | | to 51.3%). | The | LLM’s | internal | | | | | | | | |
| -------- | --- | ------------- | --- | ---------- | --- | ----- | -------- | --------- | --- | -------------- | ---- | ----- | --- | ---- | --------- |
| | | | | | | | | detectors | and | REVA on larger | open | LLMs. | We | will | also test |
representations thus carry information that neither its verdict whether reading the internal representations after a short self-
northestatic-analysisfindingsprovidealone.Withintheprobe,
| | | | | | | | | generated | analysis | adds information. | | Success | criterion: | | REVA |
| --------- | --------- | ------- | --- | ---------- | ------- | --- | --------- | --------- | ---------- | ----------------- | ------ | -------- | ---------- | ----- | ------ |
| replacing | attention | pooling | | by a plain | average | of | the token | | | | | | | | |
| | | | | | | | | matches | or exceeds | them | on the | same LLM | at | every | scale. |
states lowers accuracy by 4.2 points, which suggests that the Step 2: Calibrated screening in coding agents. We will
| evidence | of | a vulnerability | lies | in a | few tokens. | Using | a single | | | | | | | | |
| -------- | --- | --------------- | ---- | ---- | ----------- | ----- | -------- | --- | --- | --- | --- | --- | --- | --- | --- |
learnaproject-andfunction-dependentthreshold,andintegrate
layer, omitting the standardization or dropping the ranking REVA into a coding-agent loop that escalates only high-risk
| term | lowers | accuracy | by 0.4 | to 1.5 | points, | and | removing | | | | | | | | |
| -------------- | ------ | -------- | ---------- | ------ | -------- | -------- | -------- | --------- | ----- | ---------------- | --- | ---- | ---------------------- | --- | --- |
| | | | | | | | | functions | to an | agentic detector | | or a | proof-of-vulnerability | | |
| the repository | | context | or keeping | | only the | function | by 1.0 | | | | | | | | |
check[32].Successcriterion:foragivenbudgetofescalations,
and 1.9 points, none of which is significant. On PrimeVul’s the loop finds more vulnerabilities than the agentic detector
| short | functions, | the function | | alone | thus | carries | most of the | | | | | | | | |
| ----- | ---------- | ------------ | --- | ----- | ---- | ------- | ----------- | --- | --- | --- | --- | --- | --- | --- | --- |
alone.
signal;weexpectthecontexttomattermoreforinterprocedural Step3:GeneralizationbeyondPrimeVul.Wewillevaluate
vulnerabilities.
| | | | | | | | | on other | datasets | [33]–[35], | on | interprocedural | | vulnerabil- | |
| ------- | --- | --------- | --- | -------------- | --- | ---- | ----------- | ----------- | -------- | ---------------------- | --- | --------------- | --------- | ----------- | ---- |
| Leakage | and | selection | on | the evaluation | | data | can inflate | | | | | | | | |
| | | | | | | | | ities [32], | [36] | and on vulnerabilities | | | disclosed | after | each |
probing results [21], [31]. When no project appears in both LLM’s training cutoff, and study fine-tuning the LLM with the
| training | and | evaluation | folds, | cross-validation | | accuracy | drops | | | | | | | | |
| -------- | --- | ---------- | ------ | ---------------- | --- | -------- | ----- | --------------- | --- | ------------------ | --- | ---- | ----- | ------------ | --- |
| | | | | | | | | same objective. | | Success criterion: | | REVA | keeps | its accuracy | on |
by only 1.3 to 2.7 points, so most of what the probe learns vulnerabilities the LLM cannot have seen.
transferstounseenprojects.Likewise,choosingthefourprobed
layers on the training folds only, instead of on all development VI. CONCLUSION
data, changes accuracy by less than half a point, so the layer REVA detects vulnerabilities by probing an LLM’s internal
choice does not inflate the results. Accuracy is, however, 6 representationsofafunctionanditssecuritycontextinasingle
| points | lower | on the test | set | than in | cross-validation | | (60.0% | | | | | | | | |
| ------ | ----- | ----------- | --- | ------- | ---------------- | --- | ------ | ------- | ----- | ------------- | -------- | --- | --------- | ---------- | --- |
| | | | | | | | | forward | pass. | On PrimeVul’s | held-out | | test set, | it matches | or |
against 66.2% for GLM-5.3-Flash). As PrimeVul splits its data exceeds agentic detectors run on the same LLM, with fewer
chronologically, this gap likely reflects the shift towards newer false positives, in about one second per function and at a small
code, which cross-validation does not capture. fraction of their cost. An LLM’s internal representations thus
| | | | | | | | | carry more | information | about | vulnerabilities | | than | its | verdicts, |
| ------ | --- | -------- | ----- | ------------- | --- | -------- | ------- | ---------- | ----------- | ----- | --------------- | --- | ---- | --- | --------- |
| Answer | to | RQ3: The | probe | is essential. | | A direct | verdict | | | | | | | | |
from the same LLM falls to chance, and replacing attention and reading them enables real-time screening of every function
as it is written.
| pooling | by | an average | of the | token | states | costs | 4.2 points, | | | | | | | | |
| --------- | --- | ---------- | ------- | ----- | -------------- | --------- | ----------- | --- | --- | --- | --- | --- | --- | --- | --- |
| whereas | the | repository | context | has | no significant | | effect on | | | | | | | | |
| PrimeVul. | | Accuracy | largely | holds | on unseen | projects. | | | | | | | | | |
REFERENCES [23] A.Khatri,“Probingtheprefill:Detectingcodevulnerabilitiesvialatent
activations,”arXivpreprintarXiv:2608.16970,2026.
[1] B. Boehm and V. R. Basili, “Software defect reduction top 10 list,” [24] A.Sternfeld,A.Kucharavy,andL.Dolamic,“Minimalpromptpertur-
Computer,vol.34,no.1,pp.135–137,2001. bationsleadtocodevulnerabilities:Promptfragilityandhidden-state
[2] B.Johnson,Y.Song,E.Murphy-Hill,andR.Bowdidge,“Whydon’t signalsincodingLLMs,”arXivpreprintarXiv:2605.29737,2026.
software developers use static analysis tools to find bugs?” in Proc. [25] C.DiCicco,“CodecorrectnessislinearlydecodablefromLLMhidden
35thIEEE/ACMInt.Conf.onSoftwareEngineering(ICSE),2013,pp. statesbeforegeneration,”arXivpreprintarXiv:2606.14530,2026.
672–681. [26] N.RisseandM.Böhme,“Uncoveringthelimitsofmachinelearning
[3] X.Du,J.Feng,Y.Zou,W.Xu,J.Ma,W.Zhang,S.Liu,X.Peng,and forautomaticvulnerabilitydetection,”inProc.33rdUSENIXSecurity
Y.Lou,“ReducingfalsepositivesinstaticbugdetectionwithLLMs:An Symposium,2024.
empiricalstudyinindustry,”arXivpreprintarXiv:2601.18844,2026. [27] F.Weissberg,L.Pirch,E.Imgrund,J.Möller,T.Eisenhofer,andK.Rieck,
[4] Z. Wang, T. Shi, J. He, M. Cai, J. Zhang, and D. Song, “CyberGym: “LLM-basedvulnerabilitydiscoverythroughthelensofcodemetrics,”in
EvaluatingAIagents’real-worldcybersecuritycapabilitiesatscale,”in Proc.48thIEEE/ACMInt.Conf.onSoftwareEngineering(ICSE),2026.
Proc.Int.Conf.onLearningRepresentations(ICLR),2026. [28] Semgrep,Inc.,“Semgrep:Lightweightstaticanalysisformanylanguages,”
[5] DARPA,“AICyberChallenge(AIxCC),”https://www.darpa.mil/research/ https://semgrep.dev,2026.
programs/ai-cyber, 2025, Semifinal at DEF CON 2024, final at DEF [29] Meta, “Muse-Glimmer-30B,” https://huggingface.co/meta-models/
CON2025. Muse-Glimmer-30B,2026.
[6] W.Tang,X.Zhang,J.Liu,J.Xiao,X.Xiao,J.Yang,Y.Ma,Z.Liu, [30] Qwen Team, “Qwen3.8-27B,” https://huggingface.co/Qwen/Qwen3.
Z.Li,Z.Wang,W.Luo,Q.Li,L.Wang,andP.Xiangli,“VulTriage: 8-27B,2026.
Triple-pathcontextaugmentationforLLM-basedvulnerabilitydetection,” [31] J.HewittandP.Liang,“Designingandinterpretingprobeswithcontrol
arXivpreprintarXiv:2605.09461,2026. tasks,” in Proc. Conf. on Empirical Methods in Natural Language
[7] Z. Wang, G. Li, J. Li, H. Zhu, and Z. Jin, “VulAgent: Hypothesis- Processing(EMNLP),2019.
validationdrivenmulti-agentarchitectureforvulnerabilitydetection,”in [32] A.Lbath,M.Suri,A.Delaitre,V.Okun,M.-R.Amini,R.D.Sriram,and
FindingsoftheAssociationforComputationalLinguistics:ACL2026, D.Manocha,“CyberForge:Verifiedvulnerabilityinjectionatrepository
2026,pp.18598–18616. levelforcybersecurityagenttraining,”arXivpreprintarXiv:2608.06471,
[8] R.Widyasari,M.Weyssow,I.C.Irsan,H.W.Ang,F.Liauw,E.L.Ouh, 2026.
L.K.Shar,H.J.Kang,andD.Lo,“Letthetrialbegin:Amock-court [33] J. He and M. Vechev, “Large language models for code: Security
approachtovulnerabilitydetectionusingLLM-basedagents,”inProc. hardening and adversarial testing,” in Proc. ACM SIGSAC Conf. on
48thIEEE/ACMInt.Conf.onSoftwareEngineering(ICSE),2026. ComputerandCommunicationsSecurity(CCS),2023.
[9] Y.Ding,Y.Fu,O.Ibrahim,C.Sitawarin,X.Chen,B.Alomair,D.Wagner, [34] Y.Chen,Z.Ding,L.Alowain,X.Chen,andD.Wagner,“DiverseVul:A
B.Ray,andY.Chen,“Vulnerabilitydetectionwithcodelanguagemodels: newvulnerablesourcecodedatasetfordeeplearningbasedvulnerability
How far are we?” in Proc. 47th IEEE/ACM Int. Conf. on Software detection,”inProc.26thInt.Symp.onResearchinAttacks,Intrusions
Engineering(ICSE),2025. andDefenses(RAID),2023.
[10] X.-C.Wen,Y.Yang,C.Gao,Y.Xiao,andD.Ye,“Boostingvulnerability [35] A.Lbath,M.-R.Amini,A.Delaitre,andV.Okun,“AVIATOR:Towards
detectionofLLMsviacurriculumpreferenceoptimizationwithsynthetic AI-agenticvulnerabilityinjectionworkflowforhigh-fidelity,large-scale
reasoning data,” in Findings of the Association for Computational codesecuritydataset,”arXivpreprintarXiv:2508.20866,2025.
Linguistics:ACL2025,2025,pp.8935–8949. [36] A. Lbath, “Toward scalable automated repository-level datasets for
[11] O.Ovadia,E.BenZaken,E.Guttman,andO.MorenoKadosh,“MiST: softwarevulnerabilitydetection,”arXivpreprintarXiv:2603.17974,2026.
Mid-trainingLLMsforcybersecurity,”arXivpreprintarXiv:2609.18496,
2026.
[12] A.AzariaandT.Mitchell,“TheinternalstateofanLLMknowswhen
it’slying,”inFindingsoftheAssociationforComputationalLinguistics:
EMNLP2023,2023.
[13] C. Burns, H. Ye, D. Klein, and J. Steinhardt, “Discovering latent
knowledgeinlanguagemodelswithoutsupervision,”inProc.Int.Conf.
onLearningRepresentations(ICLR),2023.
[14] S. Marks and M. Tegmark, “The geometry of truth: Emergent linear
structureinlargelanguagemodelrepresentationsoftrue/falsedatasets,”
inProc.Conf.onLanguageModeling(COLM),2024.
[15] H.Orgad,M.Toker,Z.Gekhman,R.Reichart,I.Szpektor,H.Kotek,
andY.Belinkov,“LLMsknowmorethantheyshow:Ontheintrinsic
representationofLLMhallucinations,”inProc.Int.Conf.onLearning
Representations(ICLR),2025.
[16] S.Kadavath,T.Conerly,A.Askelletal.,“Languagemodels(mostly)
knowwhattheyknow,”arXivpreprintarXiv:2207.05221,2022.
[17] M.Turpin,J.Michael,E.Perez,andS.R.Bowman,“Languagemodels
don’t always say what they think: Unfaithful explanations in chain-
of-thoughtprompting,”inAdvancesinNeuralInformationProcessing
Systems(NeurIPS),2023.
[18] Z.ai,“GLM-5.3-Flash,”https://huggingface.co/zai-org/GLM-5.3-Flash,
2026.
[19] S. Ullah, M. Han, S. Pujar, H. Pearce, A. Coskun, and G. Stringhini,
“LLMscannotreliablyidentifyandreasonaboutsecurityvulnerabilities
(yet?): A comprehensive evaluation, framework, and benchmarks,” in
Proc.IEEESymposiumonSecurityandPrivacy(S&P),2024.
[20] B.Steenhoek,M.M.Rahman,M.K.Roy,M.S.Alam,H.Tong,S.Das,
E. T. Barr, and W. Le, “To err is machine: Vulnerability detection
challenges LLM reasoning,” in Proc. 47th IEEE/ACM Int. Conf. on
SoftwareEngineering(ICSE),2025.
[21] Y.Belinkov,“Probingclassifiers:Promises,shortcomings,andadvances,”
ComputationalLinguistics,vol.48,no.1,pp.207–219,2022.
[22] W. Yu, R. Mangal, T. Zhuo, M. Fredrikson, and C. S. Pasareanu, “A
mixture of linear corrections generates secure code,” arXiv preprint
arXiv:2507.09508,2025.