The model isn't cooperating

Published: Tuesday, 6 October 2026 at 17:14 UTC Updated: Tuesday, 6 October 2026 at 17:14 UTC Have you ever felt like a model isn't actually trying to do what you asked? Did you just ask it wrong, or is there something else at play here? In this post, I'll briefly explore this phenomenon, the impact, and what you can do about it.BackgroundAt Black Hat USA a couple of months ago, I published Can AI do novel security research? Meet the HTTP Terminator. This proved that AI can perform original security research and invent genuinely novel hacking techniques, but it also highlighted an area where models really struggle. A research cascade is where one discovery is used as fuel for more discoveries affecting different targets (not to be confused with bug chaining), and models consistently underperform at this process.After publication, I decided to push hard to overcome this barrier and achieve an autonomous cascade, in order to get some fresh material for the Offensive AI Con edition of my presentation. I broke my cascade methodology down into multiple steps, and threw a swarm of agents at every aspect of it, giving them the tools to test arbitrary ideas at scale so they weren't confined to the HTTP Desync research space:This process burned quite a few tokens, and only yielded one significant discovery. I was genuinely quite surprised so I took a closer look at the traces, and began to develop a suspicion.The suspicionBroadly speaking, original security research means seeking out high-impact behaviors that are hard to observe. Nobody cares about low-impact behaviors, and if something is easy to observe then someone else probably already found it so you're not really doing original research.As I analyzed the traces, it seemed like the model was steering toward low-impact behaviors that were easy to observe. The model would try to complete its objective, but use the wiggle-room in the prompt to steer in a direction that actually sabotaged its performance.Specific, highly-focused tasks like "use this HTTP desync trigger to achieve response queue poisoning on this website" worked fine since they offered minimal wiggle-room. But when given a broad prompt like "Explore other threats arising from the same root cause", the model would silently sabotage its performance.It seemed like the more ambitious and open-ended your research, the more you'd get silently sabotaged.The evalI initially observed this behavior on OpenAI's daybreak-blue which is powered by …