Cloud Security Podcast SIEM vs. Data Lake: Why We Ditched Traditional Logging?
Dec 2, 2025
Cliff Crosland, co-founder and CEO of Scanner.dev, shares his insights from his journey in transforming security data management. He discusses the high costs and challenges of traditional SIEMs, revealing how his initial attempts at building an in-house data lake hit major roadblocks. Cliff highlights issues like slow queries and the engineering lift required for usability. He also explores the potential of AI in enhancing detection engineering and offers advice on when to build or buy a data lake solution.
AI Snips
Chapters
Transcript
Episode notes
Log Volume Growth Broke SIEM Economics
- Log volumes exploded with containerization and SaaS, making traditional SIEM retention economically infeasible at scale.
- Data lakes on object storage become the practical place to keep massive log volumes long-term.
From SIEM To S3: A Cost-Driven Shift
- Cliff Crosland describes moving 90% of logs to S3 after their SIEM hit volume limits and a license increase would outstrip the engineering budget.
- The S3 + Athena approach became a "black hole" where large queries were slow and unusable for investigations.
Only Build A Lake If You Have Data Engineers
- Expect a significant data-engineering lift to ingest and normalize many log sources into SQL-style data lakes.
- Only build in-house if you have strong, ongoing data engineering resources or can share existing lake expertise across teams.