Detection at Scale Databricks' Dave Herrald on Building SOCs with Data Lakes & Focused AI Agents
Dave Herrald, Global Head of Cybersecurity GTM at Databricks, tells Jack about transforming security operations through modern data lake architectures and strategic AI implementation. He discusses the practical benefits of separating storage from compute, giving security teams direct control over data retention while maintaining operational flexibility.
The conversation explores how organizations can move beyond traditional SIEM limitations by leveraging cost-effective data lake storage with advanced analytics capabilities. They touch on AI agents in security, where Dave advocates for focused agents over broad analyst replacement approaches. He also addresses common concerns about hallucinations, framing them as engineering challenges rather than insurmountable obstacles, and shares real-world examples of successful agent implementations.
Topics discussed:
- Moving from traditional SIEM architectures to modern data lake approaches for cost-effective security analytics and data control.
- Implementing focused AI agents for specific security tasks like context gathering rather than attempting broad analyst replacement.
- Leveraging graph analytics for security operations including CMDB visualization, breach scoping, and vulnerability prioritization across enterprise environments.
- Addressing AI hallucinations through prompt engineering and proper context management rather than avoiding AI implementation entirely.
- Building detection capabilities using SQL and Python for analytics that provide supersets of traditional SIEM query languages.
- Creating normalization frameworks using standards like OCSF to enable consistent data analytics across diverse security data sources.
- Developing career resilience in security through mission-focused thinking, continuous AI learning, and building practical skills.
- Comparing modern AI agents to traditional SOAR platforms for automation effectiveness and maintenance requirements.
- Establishing data governance and access controls in security data lakes while maintaining operational flexibility and cost effectiveness.
Listen to more episodes: