Building An Internal Database As A Service Platform At Cloudflare
Aug 28, 2023
auto_awesome
This podcast explores how Cloudflare provides PostgreSQL as a service to their developers for low latency and high uptime services at global scale. They discuss challenges in maintaining high uptime and managing data volume, scaling considerations and load balancing strategies, the evolvement of database engines, differences in version upgrades between Postgres and MySQL, innovative usage and challenges in building a database platform at Cloudflare, and lessons learned in building their system.
Building a database platform at Cloudflare involves addressing challenges in multi-region environments and ensuring high uptime.
Continuous improvement, customization, and engagement with the open-source community are crucial for innovation in Postgres as a service.
Deep dives
Building a Database Platform at CloudFlare
In this podcast episode, Vignesh Ravachandran discusses his role as an engineering manager at CloudFlare, where he is responsible for building the database platform. He talks about the challenges of managing databases in a multi-region environment and the importance of maintaining high uptime due to the nature of CloudFlare's services. Vignesh also shares his insights on scaling the database workloads, addressing replication lag, and ensuring data consistency across regions. He emphasizes the need for continuous improvements in tooling and the developer experience, as well as the importance of hands-on involvement rather than relying solely on cloud providers. Additionally, he discusses the challenges and future plans for version upgrades and managing Postgres extensions. Overall, Vignesh highlights the significance of not settling for average solutions and the value of engaging with the open-source community to drive innovation in Postgres as a service.
Challenges and Lessons from Operating a Postgres Database Platform
In this podcast episode, Vignesh Ravachandran shares his experiences and insights in building and operating a Postgres database platform at CloudFlare. He discusses the unique challenges they face, such as managing databases in a multi-region, bare metal environment with high uptime requirements. Vignesh also talks about the lessons learned in optimizing performance, addressing replication lag, and scaling the database workloads. He emphasizes the importance of autonomy and the ability to customize and adapt Postgres to meet their specific needs. Vignesh highlights the value of continuous improvement, such as reducing recovery time objectives and automating upgrades. Finally, he discusses the ongoing areas of focus, including enhancing developer experience and addressing gaps in data management tooling.
Exploring the Future of Postgres as a Service
This podcast episode delves into the future of Postgres as a service. Vignesh Ravachandran discusses the evolving ecosystem around Postgres, with extensions becoming increasingly popular and Postgres being used for various purposes, including machine learning and business intelligence. He highlights the ongoing challenges in data management, such as backup and restoration, PII masking, and utilizing the same data for different purposes. Vignesh also talks about the need for improved tooling and technology for version upgrades, voice upgrades, and managing complex workloads. He emphasizes the importance of autonomy and the flexibility to build and customize solutions, as well as the vital role of the open-source community in driving innovation in the Postgres ecosystem.
Data persistence is one of the most challenging aspects of computer systems. In the era of the cloud most developers rely on hosted services to manage their databases, but what if you are a cloud service? In this episode Vignesh Ravichandran explains how his team at Cloudflare provides PostgreSQL as a service to their developers for low latency and high uptime services at global scale. This is an interesting and insightful look at pragmatic engineering for reliability and scale.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack
This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold
You shouldn't have to throw away the database to build with fast-changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. Whether it’s real-time dashboarding and analytics, personalization and segmentation or automation and alerting, Materialize gives you the ability to work with fresh, correct, and scalable results — all in a familiar SQL interface. Go to dataengineeringpodcast.com/materialize today to get 2 weeks free!
Your host is Tobias Macey and today I'm interviewing Vignesh Ravichandran about building an internal database as a service platform at Cloudflare
Interview
Introduction
How did you get involved in the area of data management?
Can you start by describing the different database workloads that you have at Cloudflare?
What are the different methods that you have used for managing database instances?
What are the requirements and constraints that you had to account for in designing your current system?
Why Postgres?
optimizations for Postgres
simplification from not supporting multiple engines
limitations in postgres that make multi-tenancy challenging
scale of operation (data volume, request rate
What are the most interesting, innovative, or unexpected ways that you have seen your DBaaS used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on your internal database platform?
When is an internal database as a service the wrong choice?
What do you have planned for the future of Postgres hosting at Cloudflare?
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
