Ep173: Simplifying Elasticsearch at Scale: How Elastic Built Their Serverless Platform

Uri Cohen reveals how Elastic transformed from managing 50,000 complex clusters to building a seamless serverless platform that eliminates operational overhead while scaling globally

Topics Include:

• Johan Broman of AWS hosts Uri Cohen who leads Elastic's platform products team
• Uri shares his nine-year journey at Elastic from small company to global scale
• Elasticsearch started 15 years ago, becoming popular for search, logs, and security events
• Elastic Cloud launched 2015, but users struggled with shards, nodes, and infrastructure complexity
• Serverless eliminates operational concerns, letting users just ingest and analyze their data
• Design goal: maintain familiar Elasticsearch experience while removing all infrastructure management burden
• Chose complete architectural redesign over retrofitting auto-scaling to existing infrastructure
• New architecture uses S3 persistence with lightweight routing layer serving 50,000+ clusters
• Cell-based design limits blast radius and improves multi-tenancy across 40+ global regions
• Learned S3 API costs can explode unexpectedly without careful request pattern optimization
• AI transforms security workflows: 10,000 alerts become 3 actionable attack summaries automatically
• Weekly continuous deployment enables faster innovation delivery without waiting for version releases

Participants:

• Uri Cohen – Vice President of Product Management, Platform, Elastic
• Johan Broman – EMEA ISV Head of Solutions Architecture, Amazon Web Services

See how Amazon Web Services gives you the freedom to migrate, innovate, and scale your software company at https://aws.amazon.com/isv/