Amazon S3: The Backbone of Modern Data Systems

49 snips

Jun 3, 2025

Mai-Lan Tomsen Bukovec, Vice President of Technology at AWS, reveals the remarkable evolution of Amazon S3, a key data repository since 2006. She shares fascinating insights on how S3 transformed from basic storage into the backbone of modern data architecture, enabling scalable data lakes. Discussion includes the importance of metadata, the integration of S3 with Iceberg, and innovations like strong consistency. Companies like Adobe and Netflix leverage S3 for efficiency, showcasing its role in navigating both structured and unstructured data challenges.

Ask episode

AI Snips

Chapters

Books

Transcript

Episode notes

ANECDOTE

Mai-Lan's Journey to S3 Leadership

Mai-Lan Tomsen Bukovec joined AWS in 2010 initially working in compute before joining S3 in 2013 as general manager.
Netflix was an early S3 data lake user, using it as their single source of truth for all data systems.

INSIGHT

S3 Enabled Modern Data Patterns

The modern data pattern started in 2006 with S3 and Hadoop enabling large shared data sets accessed by varied compute workloads.
S3 evolved to support concurrency, durability, cost-effectiveness, and strong consistency driven by early adopters like Netflix and Pinterest.

ANECDOTE

FINRA's Massive Bursty Workload on S3

FINRA uses S3 to regulate U.S. stock market trades, processing up to 906 billion records per day within strict SLAs.
Their bursty workload style highlights why S3's scale and resource sharing are uniquely suited for massive data ingestion and analytics.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Summary
In this episode of the Data Engineering Podcast Mai-Lan Tomsen Bukovec, Vice President of Technology at AWS, talks about the evolution of Amazon S3 and its profound impact on data architecture. From her work on compute systems to leading the development and operations of S3, Mylan shares insights on how S3 has become a foundational element in modern data systems, enabling scalable and cost-effective data lakes since its launch alongside Hadoop in 2006. She discusses the architectural patterns enabled by S3, the importance of metadata in data management, and how S3's evolution has been driven by customer needs, leading to innovations like strong consistency and S3 tables.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
This is a pharmaceutical Ad for Soda Data Quality. Do you suffer from chronic dashboard distrust? Are broken pipelines and silent schema changes wreaking havoc on your analytics? You may be experiencing symptoms of Undiagnosed Data Quality Syndrome — also known as UDQS. Ask your data team about Soda. With Soda Metrics Observability, you can track the health of your KPIs and metrics across the business — automatically detecting anomalies before your CEO does. It’s 70% more accurate than industry benchmarks, and the fastest in the category, analyzing 1.1 billion rows in just 64 seconds. And with Collaborative Data Contracts, engineers and business can finally agree on what “done” looks like — so you can stop fighting over column names, and start trusting your data again.Whether you’re a data engineer, analytics lead, or just someone who cries when a dashboard flatlines, Soda may be right for you. Side effects of implementing Soda may include: Increased trust in your metrics, reduced late-night Slack emergencies, spontaneous high-fives across departments, fewer meetings and less back-and-forth with business stakeholders, and in rare cases, a newfound love of data. Sign up today to get a chance to win a $1000+ custom mechanical keyboard. Visit dataengineeringpodcast.com/soda to sign up and follow Soda’s launch week. It starts June 9th.
Your host is Tobias Macey and today I'm interviewing Mai-Lan Tomsen Bukovec about the evolutions of S3 and how it has transformed data architecture

Interview

Introduction
How did you get involved in the area of data management?
Most everyone listening knows what S3 is, but can you start by giving a quick summary of what roles it plays in the data ecosystem?
What are the major generational epochs in S3, with a particular focus on analytical/ML data systems?
- The first major driver of analytical usage for S3 was the Hadoop ecosystem. What are the other elements of the data ecosystem that helped shape the product direction of S3?
Data storage and retrieval have been core primitives in computing since its inception. What are the characteristics of S3 and all of its copycats that led to such a difference in architectural patterns vs. other shared data technologies? (e.g. NFS, Gluster, Ceph, Samba, etc.)
How does the unified pool of storage that is exemplified by S3 help to blur the boundaries between application data, analytical data, and ML/AI data?
What are some of the default patterns for storage and retrieval across those three buckets that can lead to anti-patterns which add friction when trying to unify those use cases?
The age of AI is leading to a massive potential for unlocking unstructured data, for which S3 has been a massive dumping ground over the years. How is that changing the ways that your customers think about the value of the assets that they have been hoarding for so long?
- What new architectural patterns is that generating?
What are the most interesting, innovative, or unexpected ways that you have seen S3 used for analytical/ML/Ai applications?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on S3?
When is S3 the wrong choice?
What do you have planned for the future of S3?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA