Evaluating AI Model Performance: The Importance of a Structured Approach

Guest:

• Dominik Swierad,  Senior PM D&R AI and Sec-Gemini

Topics:

• When introducing AI agents to security teams at Google, what was your initial strategy to build trust and overcome the natural skepticism? Can you walk us through the very first conversations and the key concerns that were raised?
• With a vast array of applications, how did you identify and prioritize the initial use cases for AI agents within Google's enterprise security? 
• What specific criteria made a use case a good candidate for early evaluation? Were there any surprising 'no-go' areas you discovered?"
• Beyond simple efficiency gains, what were the key metrics and qualitative feedback mechanisms you used to evaluate the success of the initial AI agent deployments? 
• What were the most significant hurdles you faced in transitioning from successful pilots to broader adoption of AI agents?
• How do you manage the inherent risks of autonomous agents, such as potential for errors or adversarial manipulation, within a live and critical environment like Google's?
• How has the introduction of AI agents changed the day-to-day responsibilities and skill requirements for Google's security engineers? 
• From your unique vantage point of deploying defensive AI agents, what are your biggest concerns about how threat actors will inevitably leverage similar technologies?

Resources:

• EP235 The Autonomous Frontier: Governing AI Agents from Code to Courtroom
• EP236 Accelerated SIEM Journey: A SOC Leader's Playbook for Modernization and AI
• EP224 Protecting the Learning Machines: From AI Agents to Provenance in MLSecOps
• EP227 AI-Native MDR: Betting on the Future of Security Operations?
• EP75 How We Scale Detection and Response at Google: Automation, Metrics, Toil