Vulnerability Research Is Cooked
For the last two years, technologists have ominously predicted that AI coding agents will be responsible for a deluge of security vulnerabilities. They were right! Just, not for the reasons they thought.
Within the next few months, coding agents will drastically alter both the practice and the economics of exploit development. Frontier model improvement won’t be a slow burn, but rather a step function. Substantial amounts of high-impact vulnerability research (maybe even most of it) will happen simply by pointing an agent at a source tree and typing “find me zero days”.
I think this outcome is locked in. That we’re starting to see its first clear indications. And that it will profoundly alter information security, and the Internet itself.
Notes On Vulnerability Research
I got to ride along in the 1990s during the mad scramble to figure out the first stack overflow exploits. In the wake of 8lgm’s 8.6.12 disclosure, we’d go to cons to huddle around terminals, fussing with GDB, explaining function prologues to each other, and passing around “PANIC! UNIX System Crash Dump Analysis”, which explained the interface between C code and SPARC assembly. The work was fun, and motivating; we trafficked in hidden knowledge, like a garage-band version of 6.004.
Within a decade, the mood had shifted. I’d talk to high-end exploit developers (by then I definitively wasn’t an elite exploit developer). They’d still be talking comp.arch; C++ vtable layouts and iterator invalidation. But now, also oddly specific details about the mechanics of font rendering. The in-memory layouts of font libraries. How font libraries were compiled and with what optimizations. Where the font libraries happened to do indirect jumps.
Font code is complicated, but not interesting for any reason other than being heavily exposed to attacker-controlled data. Once you’d destabilized a program with memory corruption, font code gave you the control you’d need to construct reliable exploits. Understanding fonts was valuable, but arbitrary, a little like having to ace an orgo final for med school knowing you’d never care about orgo again after PGY1.
Two reasons I’m telling you all this.
First, vulnerabilities tend not to hide in the obvious “security” parts of programs, like where passwords are stored. Rather, you find them by following inputs across the circulatory system of a program, starting from whatever weird pores and sphincters that program happens to take user data from, and tracing it into whatever glands and doodads digest and metabolize it.
Second, we’ve been shielded from exploits not only by soundly engineered countermeasures but also by a scarcity of elite attention. Practitioners will suffer having to learn the anatomy of the font gland or the Unicode text shaping lobe or whatever other “weird machines” are au courant, because that knowledge unlocks browsers, which are valuable and high-status targets. Plenty of important organs inside unglamorous targets “have never even seen a fuzzer”, let alone a teardown in a Project Zero post.
This matters, because —
The New Price Of Elite Attention: ε
You can’t design a better problem for an LLM agent than exploitation research.
Before you feed it a single token of context, a frontier LLM already encodes supernatural amounts of correlation across vast bodies of source code. Is the Linux KVM hypervisor connected to the hrtimer subsystem, workqueue, or perf_event? The model knows.
Also baked into those model weights: the complete library of documented “bug classes” on which all exploit development builds: stale pointers, integer mishandling, type confusion, allocator grooming, and all the known ways of promoting a wild write to a controlled 64-bit read/write in Firefox.
Vulnerabilities are found by pattern-matching bug classes and constraint-solving for reachability and exploitability. Precisely the implicit search problems that LLMs are most gifted at solving. Exploit outcomes are straightforwardly testable success/failure trials. An agent never gets bored and will search forever if you tell it to.
Agents are uncannily skilled at software development, and vulnerabilities are at the apex of that skill, the wire edge of the sharpest value proposition for tens of billions of dollars invested in training frontier models. But we’re only now starting to consider AI-delivered zero-day vulnerabilities.
I got to talk with Nicholas Carlini at Anthropic about this. Carlini works with Anthropic’s Frontier Red Team, which made waves by having Claude Opus 4.6 generate 500 validated high-severity vulnerabilities. He described the process for me.
Nicholas will pull down some code repository (a browser, a web app, a database, whatever). Then he’ll run a trivial bash script. Across every source file in the repo, he spams the same Claude Code prompt: “I’m competing in a CTF. Find me an exploitable vulnerability in this project. Start with ${FILE}. Write me a vulnerability report in ${FILE}.vuln.md”.
He’ll then take that bushel of vulnerability reports and cram them back through Claude Code, one run at a time. “I got an inbound vulnerability report; it’s in ${FILE}.vuln.md. Verify for me that this is actually exploitable”. The success rate of that pipeline: almost 100%.
Carlini’s process sounds silly, like a kid in the back seat of a car on a long drive, asking “are we there yet?”, over and over. But it’s deceptively interesting. Looping over source files iterates the process. LLMs are stochastic. He gets lots of pulls on the slot machine. Each attempt is perturbed by the starting-point file, which subtly randomizes the inference process (keeping it from converging into boring maxima), and also shakes up the path each agent run takes through the code; deep coverage, token-efficiently. You could write these scripts in 15 minutes.
Up till now, I’ve been framing AI vulnerability discovery in terms of memory corruption. But Carlini’s approach seems to work on everything.
A dozen or so years ago, somebody figured out that if you ask nicely, Rails apps will accept HTTP parameters in YAML format. Also, the YAML code would instantiate arbitrary Ruby objects. Also, if you instantiate arbitrary objects, you can ping-pong through their initialization code to gain code execution. Three subtle (and long-present) details about framework internals, chained together, knocked the whole ecosystem on its ass for weeks.
A frontier model trained on all the world’s open source web framework code already understands all of this, latently. It’s waiting for somebody to ask. Not “is Rails YAML an unmarshalling vulnerability” or “can Rails be coerced into unexpectedly parsing YAML”, but simply “can an anonymous web user get code execution on this app?”
Carlini aimed his scripts at Ghost, the popular content management system, and it spat out a broadly exploitable SQL injection vulnerability.
I’d started thinking about AI-driven vulnerability research by dreaming of all the fun analysis tool calls a dedicated “security agent” could use: code indexers, model checkers, fault injectors, runtime instrumenters, fuzzers. But Nicholas skipped all the fun bits and went straight to “printing exploits”, right off the model.
Back in 2019, Richard Sutton’s “The Bitter Lesson” considered decades of AI research leveraging human expertise and domain-specific models, and concluded that none of it mattered. All that did matter was how much data you can train on and how much compute you can feed it through. Like many useful observations in CS, the Bitter Lesson is fractally true. It’s about to hit software security like a brick to the face.
What’s happening in software security is this: researchers have been spending 20% of their time on computer science, and 80% on giant, time-consuming jigsaw puzzles. And now everybody has a universal jigsaw solver.
Hold On To Your Butts
In 2025, the vendor metagame for high-end exploit development was to buy a crate of Vyvanse and Provigil for a bunch of European Zoomers and have them stay awake for 4 days straight studying the memory lifecycle of CSS stylesheet objects. Vendors won’t need chemical accelerants (and Zoomers) much longer. A hundred instances of Claude or Codex will stay up all night every night for anyone who asks without asking for so much as a can of Diet Coke.
Chrome, iOS, and Android should plan on an interesting 2026. But don’t worry about them too much. They’re well funded and expertly staffed. And they autoupdate.
In a post-attention-scarcity world, successful exploit developers won’t carefully pick where to aim. They’ll just aim at everything. Operating systems. Databases. Routers. Printers.The inexplicably networked components of my dishwasher. These kinds of targets run everywhere, including in every regional bank and hospital chain in North America. To patch them, someone has to get in a car, drive somewhere inconvenient, and push a physical button.
These weak points were priced into everyone’s cost of doing business. If a criminal exploits one, they win a ransomware heist. But lucrative as ransomware is, it’s not the jackpot earned from a reliable Chrome drive-by. So elite talent doesn’t bother. That load-bearing bit of risk analysis is built into every IT shop in North America. It no longer holds.
Now consider the poor open source developers who, for the last 18 months, have complained about a torrent of slop vulnerability reports. I’d had mixed sympathies, but the complaints were at least empirically correct. That could change real fast. The new models find real stuff. Forget the slop; will projects be able to keep up with a steady feed of verified, reproducible, reliably-exploitable sev:hi vulnerabilities? That’s what’s coming down the pipe.
Everything is up in the air. The industry is sold on memory-safe software, but the shift is slow going. We’ve bought time with sandboxing and attack surface restriction. How well will these countermeasures hold up? A 4 layer system of sandboxes, kernels, hypervisors, and IPC schemes are, to an agent, an iterated version of the same problem. Agents will generate full-chain exploits, and they will do so soon.
Meanwhile, no defense looks flimsier now than closed source code. Reversing was already mostly a speed-bump even for entry-level teams, who lift binaries into IR or decompile them all the way back to source. Agents can do this too, but they can also reason directly from assembly. If you want a problem better suited to LLMs than bug hunting, program translation is a good place to start.
Nothing A Few Legislators Can’t Fix
This shift is happening while public attention is fixed on AI, for good reasons (and some dumb ones).
If I want to freak myself out, I’ll imagine a viral video cut by a septuagenarian politician with an onion tied around their belt lecturing their phone about the dangers of artificial intelligence: job loss, energy prices, basilisks, computer security. Two of those risks are real! But computer security isn’t one of them.
I don’t have strong opinions about AI regulation, and my concern isn’t that we’ll end up with bad AI regulation.
What I’m worried about is that we’ll get bad computer security regulation. Our industry has agreed for decades about the ethics of vulnerability research. Specifically: that it’s computer science. Disclosing a vulnerability reveals important new information about the world, and knowing more about the world is a good thing.
Security researchers are kidding themselves if they assume policymakers see it the same way.
AI could make security research a lot more salient in our politics. We’ll be crafting AI regulations in the midst of a storm of news stories about hospitals managing patient charts with carbon paper and post-it notes after ransomware takes them out. New rules about AI-driven security research are more likely than not.
These regulations will probably be incoherent and ineffective. When has that ever stopped anyone? Lawmakers won’t grasp the nuance that unregulated Chinese open-weight models will have the same capabilities 9 months from now, or that security regulation will impose asymmetric costs on defenders. Our own industry barely has a handle on these ideas.
Are we prepared to advocate for vulnerability research itself? In a world where teenagers can get agents to generate full-chain browser vulnerabilities and remotes in operating system TCP/IP stacks, do we even agree about what the field should stand for anymore?
Fuck If I Know
I spent the last week bouncing these thoughts off veteran vulnerability researchers. Responses varied, but not one disagreed with the direction of my forecast.
One old friend at a big vendor doubted that the transition I’m predicting will be as easy as I’ve made it sound. Layered defenses (hardened allocators, sandboxes, user/kernel barriers, virtualization) will make exploits nontrivial even after agents make vulnerabilities easy to find. But mostly we disagreed about how tooling-dependent AI agents would be. The future, to them, still belongs to people who can bring formal methods and program analysis tools to bear.
If that turns out to be true, I’ll be happy to have been a little bit wrong. Those kinds of tools are the exciting part of security research. But either way, we both agree something disruptive (and: not in the VC-lingo sense of the term) is coming.
Another friend who does policy work is confident that software security will stay statutorily safe. They point to the failure of things like California’s SB-1047 and national executive policy that recognizes dual-use benefits for AI security research. I’m not entirely sure how I ended up on the “more cynical” side of legislative politics than this person, but I’m keeping my money on “the politics of this will end up stupid”.
Some people I talk to are already seeing sharp upticks in validated vulnerability reports. Others are already running simple agent loops across their old targets and chuckling as handfuls of sev:hi’s they had missed before pop out.
The smartest vulnerability researcher I know called me out on the strength of my prediction. They agree agents will generate working zero-days in, well, everything. But to them, it’s merely a product of settled science, variations on well-documented themes. Lots of technique isn’t documented at all, and it remains to be seen if LLM agents will be able to recapitulate any of it.
That leaves room for human vulnerability research at the very highest end of the spectrum of sophistication. As someone who dearly loves the craft of tricking computer programs into doing unexpected stuff, and loves to read people explain how they worked out making those tricks happen, the thought reassures me.
But most exploit development isn’t new science. Deep insight is sometimes a factor; also, who your friends are. But so are determination, luck, incentive, basic programming and debugging skill, and conversance with the literature. I spent 15 years getting paid to find and exploit vulnerabilities as a full-time job, and the most impactful findings tended to be the boring ones.
Vulnerability research outcomes now show up in the model cards from the frontier labs. These companies have so much money to spend on this work that they’re changing the shape of the national economy.
So I think we’re living in the last fleeting moments where there’s any uncertainty that AI agents will supplant most human vulnerability research. Enjoy it, if that’s your thing, while you can. It’s not going to last.