S3 and the Evolution of Storage with Andy Warfield

Andy Warfield joins Corey in this episode to discuss the evolution of storage technology at Amazon. This includes the evolution of S3 from archival storage to supporting modern AI and analytics. As Vice President and Distinguished Engineer at AWS, Andy is able to explain performance-enhancing innovations like S3 Tables and Common Runtime (CRT). On the other hand, challenges like compaction and namespace structuring are discussed. Reflecting on his journey from working on the Xen hypervisor to AWS, Andy shares insights into scaling S3, including buckets spanning millions of hard disks. 

Show Highlights

(0:00) Intro

(1:09) The Duckbill Group sponsor read

(1:43) Andy’s background

(3:38) How AWS envisioned services being used vs. what customers actually do with them

(6:54) The frustration of legacy applications not keeping up with the times

(10:14) Why S3 is so accurate

(15:29) S3 as a role model for how a service should be run

(18:04) The Duckbill Group sponsor read

(18:46) Why AWS made Iceberg into a native offering

(23:50) Why S3 Tables is slightly more expensive

(28:23) How Andy handled the transition from Zen to Nitro

(32:22) What Andy is currently excited about 

About Andy Warfield

Andrew Warfield is a VP / Distinguished Engineer at Amazon. As a senior technical leader at one of the world's largest technology companies, he plays a crucial role in shaping Amazon's engineering strategies and initiatives. 

Links

• LinkedIn: https://www.linkedin.com/in/andywarfield/ 
• Email: warfield@Amazon.com 

Sponsor

The Duckbill Group: duckbillgroup.com