Screaming in the Cloud
S3 and the Evolution of Storage with Andy Warfield
Feb 4, 2025
Andy Warfield, Vice President and Distinguished Engineer at Amazon Web Services, discusses the groundbreaking evolution of storage technology. He highlights how S3 has transformed from archival storage to a powerhouse for modern AI and analytics. The conversation covers innovative features like S3 Tables and the Common Runtime (CRT), along with challenges such as namespace structuring. Andy shares insights from his journey in tech, revealing how S3's scalability now employs millions of hard disks, while reflecting on the frustrations of legacy applications.
36:39
Episode guests
AI Summary
AI Chapters
Episode notes
Podcast summary created with Snipd AI
Quick takeaways
- The evolution of S3 reflects a shift from basic storage to advanced analytics capabilities, incorporating formats like Iceberg for enhanced data management.
- Amazon's adaptability to customer feedback has driven innovations in S3's functionality and API design, ensuring they meet evolving use cases effectively.
Deep dives
The Evolution of Data Lakes and Performance Optimization
Data lakes built on S3 have seen significant interest from customers, particularly regarding Parquet performance optimizations. Over the years, a transition occurred as users shifted from Parquet to more sophisticated formats like Iceberg and Open Table formats, indicating a demand for better data management solutions. As these formats gained traction, it became clear that customers were seeking enhancements in their data analytics and storage capabilities, prompting developments like the S3 Tables product. This evolution reflects the changing landscape of how organizations are leveraging cloud storage for complex analytics workloads.
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
