In this discussion, Marc Brooker, a distinguished engineer at AWS, shares insights from his extensive career transforming cloud computing. He dives into the creation of Aurora Distributed SQL, emphasizing its resilience and serverless nature. The choice of PostgreSQL as a foundation is highlighted, focusing on its robust features. Marc also explains the complexities of managing distributed databases, including concurrency control and the architecture's fault tolerance. Innovative elements like Firecracker VMs are discussed for their role in enhancing scalability and performance.
01:14:55
forum Ask episode
web_stories AI Snips
view_agenda Chapters
auto_awesome Transcript
info_circle Episode notes
question_answer ANECDOTE
Origin of Aurora D-SQL
Marc Brooker's journey at AWS influenced Aurora D-SQL's development.
Customer conversations about serverless challenges and maximizing resilience drove the project.
insights INSIGHT
Why PostgreSQL?
Aurora D-SQL leverages PostgreSQL for its mature SQL engine and extensibility.
This approach minimizes the effort required for customer adoption and allows AWS to build upon a robust foundation.
insights INSIGHT
Snapshot Isolation
Snapshot isolation in Aurora D-SQL offers a practical balance between performance and data integrity.
It's suitable for distributed systems and read-heavy applications due to its focus on write conflicts.
Get the Snipd Podcast app to discover more snips from this episode
The GeekNarrator memberships can be joined here: https://www.youtube.com/channel/UC_mGuY4g0mggeUGM6V1osdA/join
Membership will get you access to member only videos, exclusive notes and monthly 1:1 with me.
Here you can see all the member only videos: https://www.youtube.com/playlist?list=UUMO_mGuY4g0mggeUGM6V1osdA
------------------------------------------------------------------------------------------------------------------------------------------------------------------
About this episode:
------------------------------------------------------------------------------------------------------------------------------------------------------------------
In this episode of the Geek Narrator podcast, host Kaivalya Apte interviews Marc Brooker, a distinguished engineer at AWS, about Aurora D-SQL. They discuss Marc's journey at AWS, the evolution of Aurora D-SQL, and the customer-centric approach that led to its development.
Marc explains the choice of PostgreSQL as the foundation for DSQL, the architecture of the database, and the importance of snapshot isolation and concurrency control. The conversation goes into the technical aspects of DSQL, including the write process and how atomicity is maintained, providing listeners with a comprehensive understanding of this innovative database solution. This conversation also goes deep into the intricacies of database design, focusing on fault tolerance, replication strategies, and the role of Firecracker VMs in enhancing scalability. Marc Brooker discusses the architecture of Aurora D-SQL, emphasizing the importance of transaction management, the challenges of active-active deployments, and the trade-offs involved in database design. The discussion also highlights various use cases for Aurora DSQL, including its suitability for micro-services and serverless architectures, while addressing scenarios where it may not be the best fit.
Chapters
00:00 Introduction to Aurora DSQL and Marc Brooker's Journey
03:38 The Evolution of Aurora DSQL at AWS
09:24 Customer-Centric Development and Technological Enablers
12:50 Why PostgreSQL? The Choice Behind DSQL
16:39 High-Level Architecture of DSQL
22:07 Understanding Snapshot Isolation and Concurrency Control
28:45 The Write Process and Atomicity in DSQL
38:50 Designing Fault Tolerance in Databases
47:38 Replication and Transaction Commit Strategies
54:35 Active-Active Deployment and Fault Tolerance
01:00:14 Role of Firecracker VM in Scalability
01:09:27 Use Cases and Trade-offs of Aurora D-SQL
Marc's Blog: https://brooker.co.za/blog/
Marc on Aurora DSQL : https://brooker.co.za/blog/2024/12/03/aurora-dsql.html
AWS's documentation on Aurora DSQL : https://aws.amazon.com/rds/aurora/dsql/features/
------------------------------------------------------------------------------------------------------------------------------------------------------------------
Like building real stuff?
------------------------------------------------------------------------------------------------------------------------------------------------------------------
Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription.
https://app.codecrafters.io/join?via=geeknarrator
------------------------------------------------------------------------------------------------------------------------------------------------------------------
Link to other playlists. LIKE, SHARE and SUBSCRIBE
------------------------------------------------------------------------------------------------------------------------------------------------------------------
If you like this episode, please hit the like button and share it with your network.
Also please subscribe if you haven't yet.
Database internals series: https://youtu.be/yV_Zp0Mi3xs
Popular playlists:
Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-
Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17
Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d
Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN
Stay Curios! Keep Learning!
#sql #postgres #databasesystems #aws #awsdevelopers #spanner #google #cockroachdb #yugabytedb #cap #scalability #WAL #DistributedSystems #Cloud #aurora
The GeekNarrator