blogGoogle Project Zero·8 Sept 2026·Jann Horn

Testing race conditions with memory access tracing and stack-based delay injection

Project Zero confirms race conditions by tracing memory accesses and injecting delays at chosen call stacks to force bad interleavings.

Read original on projectzero.google

vulnerability-research software-testing fuzzing

Depth 5/5Score 4.57

Extract

Many security bugs are race conditions, where multi-threaded execution has to occur with the right interleaving for a negative effect to appear. This creates challenges for several use cases: Confirming bug candidates that have been discovered manually or through static analysis. Regression tests: After fixing a race condition bug, there is often no good way to write a regression test that reliably triggers the bug as part of a test suite. Automatic bug discovery, such as fuzzing: It is hard for a fuzzer to exercise all interesting interleavings of concurrent operations, or reach code paths that are only exercised when operations are racing. I mostly discover bugs by manually reading code. When I think I’ve found a bug, I normally write a test case to either prove or disprove that the bug exists. For race condition bugs, it can be hard to achieve either outcome. For Linux kernel bugs, I often resort to recompiling the kernel after adding conditional mdelay() calls (which spinloop for roughly the specified amount of time) in appropriate places; I usually make these conditional based on the name of the running thread, though sometimes more complex conditions are needed. On platforms that support DTrace (like macOS and Windows ), it is possible to use DTrace probes that call chill() for similar effect, though the utility of this is limited as DTrace can only trace on non-inline function boundaries or explicit trace points, rather than on every instruction. Regardless of platform, this approach can be time consuming and can require trial and error to definitely determine whether code is buggy. Additionally, in the Linux kernel, fixes for race condition bugs are often accompanied by hand-written ASCII diagrams showing problematic thread interleavings with call graphs and relevant memory accesses (for example, see this recent rt_spin_unlock UAF fix , or this recent jbd2 deadlock fix ). It would be convenient to have developer tooling that can analyze potentially vulnerable code and show results in a similar representation. Summary I wrote tools for exploring possible interleavings of multi-threaded test cases for the Linux kernel: A tool that automatically tests all possible A-B-A interleavings of a test case. A terminal UI for manual exploration of possible interleavings. A GUI for manual exploration of possible interleavings. The kernel part of this is intended to also be usable for discovering race conditions via fuzzing, but userspace tooling for that still …

Explained

What happened

Project Zero built MAccConc (Memory Access Concurrency), tooling to make Linux kernel race conditions reproducible instead of relying on luck and hand-placed delays.

Race bugs only appear with a specific thread interleaving, which makes them hard to confirm, hard to write regression tests for, and hard for fuzzers to hit.
The old manual approach was recompiling the kernel with mdelay() spin-loops in suspected spots, keyed on thread name, or using DTrace chill(). Slow, and limited to function boundaries.
MAccConc instead traces every memory access using KASAN outline instrumentation (helper call per access), fed to userspace through the existing KCOV channel. Compiler flags disable ASAN merging of adjacent accesses and enable hooks on globals.
Following the SKI method, it finds communication points: pairs of accesses on two threads to overlapping memory where at least one is a write. Those are the places an interleaving can matter.
It identifies each access stably across runs with count-augmented stack traces (stack trace plus how many times that site was hit), so the same access in memcpy or a reallocated object can be targeted run to run.
It then drives A-B-A interleavings automatically, plus a terminal and GUI for manual exploration, producing the kind of interleaving diagram kernel developers draw by hand.

Why it works

The insight is that you do not need to explore all interleavings, only the ones where two threads touch the same memory with a write involved. Reducing the schedule space to communication points makes exhaustive A-B-A testing tractable. Stable per-access identifiers are the enabling trick: without them you cannot ask "pause thread A right before this specific access" repeatably when addresses change each run.

The bigger pattern

This is deterministic concurrency testing by controlled scheduling, the same family as rr, ThreadSanitizer, and model checkers like CHESS. The shared idea: turn a probabilistic bug into a deterministic one by taking control of the scheduler and enumerating only the orderings that can possibly interact. The happens-before and communication-point concepts apply far beyond the kernel.

What it means in practice

Treat a race condition as reproducible, not mystical. Before fixing, build a harness that can force the bad interleaving; then it doubles as a regression test that actually fails on the old code.
Use scheduler-controlling tools where you can: ThreadSanitizer for data races in C, C++ and Go, rr for record and replay, and fault/delay injection at the exact lines two tasks share state.
Focus on communication points: shared mutable state touched by two tasks with at least one writer. That is where to add instrumentation, delays, or review attention.
For backend and distributed systems, the analogue is injecting delays and reordering at the points where requests share state (caches, counters, job queues, leader handoffs) to flush out TOCTOU and lost-update bugs, rather than hoping load tests stumble on them.
Make instrumentation stable across runs. An identifier that survives re-allocation and inlining is what lets a flaky failure become a push-button repro.