How AI guardrails are impeding the work of offensive cybersecurity researchers
AI companies' guardrails and vetted access programs meant to block malicious hackers are also hindering legitimate offensive security researchers whose job is to find vulnerabilities before criminals do.
Intelligence analysis by Llama
Researchers who probe systems for weaknesses say Anthropic and OpenAI's safety guardrails and vetted access programs treat offensive and defensive security as the same thing, when in practice the same tool is needed for both. Some are turning to unrestricted open-source models to get their work done.
AI helpers have rules that stop them from helping bad guys hack computers. But the good guys who hunt for security problems before criminals find them need that same help. The rules are like locking a toolbox because one tool could be dangerous, even though carpenters need those exact tools to build things safely.
Analysis
The Hammer Problem
Chris Anley, chief scientist at NCC Group, drew a vivid analogy in the article: an AI model that can help fix a vulnerable piece of code is, by definition, also a roadmap for finding and exploiting that vulnerability. "You can't build a house without a hammer," he said. "It's definitely a tool but it's also irreducibly a weapon as well." That duality is the heart of the dispute. AI vendors have built guardrails to prevent their models from being weaponized by criminals, but those same rails refuse to assist the people whose full-time job is to find weaknesses in software so they can be patched. Mark Dowd, a veteran zero-day broker, put the governance question bluntly: "it's not really comfortable to me that these random large companies are making arbitrary decisions about what is safe in security and what's not."
The Pull Toward Open Source
When frontier-model guardrails get in the way, several researchers told TechCrunch they fall back on open-source models that ship without restrictions. Paolo Stagno of CrowdFense said his team uses closed frontier models for reverse engineering but explicitly avoids them for vulnerability discovery or exploit development, because feeding that work into a cloud-based system risks leaking sensitive data or contaminating future training runs. For the sensitive part of the pipeline, they run open-source weights locally and keep everything in-house. An anonymous researcher at a smartphone-component maker said his company is not part of Anthropic's Cyber Verification Program, and the result is that the off-the-shelf tools are "barely useful" — if the model detects anything security-related, it stops responding. The pattern points to a quiet migration of the most sensitive offensive-security work away from the very labs that built the most capable models.
Inconsistent Rules, Patchy Coverage
Even researchers who have been approved for the looser vetted tracks report friction. Chris Thompson of RemoteThreat said the guardrails inside Anthropic's and OpenAI's programs can be inconsistent and behave differently every day, forcing practitioners to spend time negotiating with the model instead of working the problem. Giuseppe Cali, who finds zero-days for a living, said guardrails don't impede him because he deliberately keeps AI out of the actual discovery and weaponization steps, reserving it for reverse engineering and tool-building. The variety of workarounds — local open-source models, restricted use cases, sheer manual labor — suggests the current regime is not so much a clean safety boundary as a tax on productivity, one that well-resourced defenders can absorb but that pushes smaller or less credentialed researchers out of the frontier-model ecosystem entirely. The export controls on Anthropic's Mythos 5 and Fable 5 models, imposed in June and later lifted for Fable 5, added another layer of political volatility on top of the technical one.
Key points
- Anthropic and OpenAI run vetted access programs that lift some cybersecurity guardrails, but researchers say the rules remain inconsistent and the application process excludes many legitimate practitioners.
- Multiple offensive-security researchers told TechCrunch they fall back on open-source models with no guardrails when frontier tools refuse to help, particularly for exploit development and vulnerability research.
- The article highlights a dual-use dilemma: the same model output that helps fix a vulnerability also teaches an attacker how to exploit it, and vendors cannot cleanly separate the two use cases.
- Researchers also cited data-leak and training-contamination concerns as reasons to keep sensitive vulnerability work off cloud-based frontier models entirely.
If AI vendors and governments continue refining vetted-access programs like OpenAI's Trusted Access for Cyber and Anthropic's Cyber Verification Program, researchers could eventually get predictable, lower-friction access to frontier models for clearly defensive workflows, keeping sensitive vulnerability work inside accountable platforms rather than pushing it into the open-source shadows.
If guardrails remain inconsistent and vetted programs stay narrow, the most capable offensive-security researchers may increasingly rely on locally run open-source models with no oversight, while smaller firms without program access lose the productivity edge entirely, widening the gap between well-funded defenders and everyone else — and potentially leaving real vulnerabilities unexamined.
