EP251 Beyond Fancy Scripts: Can AI Red Teaming Find Truly Novel Attacks?

Nov 10, 2025

Ari Herbert-Voss, Founder and CEO of RunCybil and former security lead at OpenAI, dives into AI-powered red teaming. He discusses how Sybil automates discovery, testing, and remediation of security flaws, particularly excelling at finding tricky authentication bugs. The conversation addresses the balance of augmenting human efforts without replacing them entirely and the importance of actionable insights for development teams. Ari also shares real-world successes, showcasing how Sybil can uncover significant vulnerabilities rapidly while scaling security efforts.

Ask episode

AI Snips

Chapters

Books

Transcript

Episode notes

INSIGHT

Automating Hacker Intuition

RunCybil automates hacker intuition across discovery, testing, and remediation phases.
Automation targets repetitive checklist work so humans can focus on creative attacks.

ADVICE

Use AI To Replace Routine Pen Tests

Use AI to replace routine pen-testing and augment red teams for deeper scenarios.
Reserve human red teams for creative, high-skill assessments while automating scale tasks.

INSIGHT

Why Auth Bugs Are AI-Friendly

Authentication and authorization bugs are high-impact and hard to scan for with traditional tools.
Context-aware agents can detect IDORs and auth gaps that scripted scanners miss.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Guest:

Ari Herbert-Voss, CEO at RunSybil

Topics:

The market already has Breach and Attack Simulation (BAS), for testing known TTPs. You're calling this 'AI-powered' red teaming. Is this just a fancy LLM stringing together known attacks, or is there a genuine agent here that can discover a truly novel attack path that a human hasn't scripted for it?
Let's talk about the 'so what?' problem. Pentest reports are famous for becoming shelf-ware. How do you turn a complex AI finding into an actionable ticket for a developer, and more importantly, how do you help a CISO decide which of the thousand 'criticals' to actually fix first?
You're asking customers to unleash a 'hacker AI' in their production environment. That's terrifying. What are the 'do no harm' guardrails? How do you guarantee your AI won't accidentally rm -rf a critical server or cause a denial of service while it's 'exploring'?
You mentioned the AI is particularly good at finding authentication bugs. Why that specific category? What's the secret sauce there, and what's the reaction from customers when you show them those types of flaws?
Is this AI meant to replace a human red teamer, or make them better? Does it automate the boring stuff so experts can focus on creative business logic attacks, or is the ultimate goal to automate the entire red team function away?
So, is this just about finding holes, or are you closing the loop for the blue team? Can the attack paths your AI finds be automatically translated into high-fidelity detection rules? Is the end goal a continuous purple team engine that's constantly training our defenses?
Also, what about fixing? What makes your findings more fixable?
What will happen to red team testing in 2-3 years if this technology gets better?

Resource: