Data Cloud Cost Optimization With Bluesky Data
Data Engineering Podcast
00:00
The Tool Chain, the Data Teams and the Organizational Consequences of Data Modelling
One thing that i'm curious about, if you've gained any insight on it, is how the tool chain can contribute to either building more efficient or more wasteful pipe lines. Maybe if theyare using something like d b t, they have a better sort of corps utility of re use of data sets. If they're doing a lot of manual table definitions and work flows or custom code, that can contribute more to a wasted spend or duplication of effort. One interesting anecdote i want to share is a former head of data who is our adviser, and they were running a big snowflake instance back then. They mentioned to us recently that due to the increased cost of running snow flake based transformation
Transcript
Play full episode
Transcript
Episode notes
Summary
The latest generation of data warehouse platforms have brought unprecedented operational simplicity and effectively infinite scale. Along with those benefits, they have also introduced a new consumption model that can lead to incredibly expensive bills at the end of the month. In order to ensure that you can explore and analyze your data without spending money on inefficient queries Mingsheng Hong and Zheng Shao created Bluesky Data. In this episode they explain how their platform optimizes your Snowflake warehouses to reduce cost, as well as identifying improvements that you can make in your queries to reduce their contribution to your bill.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
- This episode is brought to you by Acryl Data, the company behind DataHub, the leading developer-friendly data catalog for the modern data stack. Open Source DataHub is running in production at several companies like Peloton, Optum, Udemy, Zynga and others. Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl
- RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
- The most important piece of any data project is the data itself, which is why it is critical that your data source is high quality. PostHog is your all-in-one product analytics suite including product analysis, user funnels, feature flags, experimentation, and it’s open source so you can host it yourself or let them do it for you! You have full control over your data and their plugin system lets you integrate with all of your other data tools, including data warehouses and SaaS platforms. Give it a try today with their generous free tier at dataengineeringpodcast.com/posthog
- Your host is Tobias Macey and today I’m interviewing Mingsheng Hong and Zheng Shao about Bluesky Data where they are combining domain expertise and machine learning to optimize your cloud warehouse usage and reduce operational costs
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Bluesky is and the story behind it?
- What are the platforms/technologies that you are focused on in your current early stage?
- What are some of the other targets that you are considering once you validate your initial hypothesis?
- Cloud cost optimization is an active area for application infrastructures as well. What are the corollaries and differences between compute and storage optimization strategies and what you are doing at Bluesky?
- How have your experiences at hyperscale companies using various combinations of cloud and on-premise data platforms informed your approach to the cost management problem faced by adopters of cloud data systems?
- What are the most significant drivers of cost in cloud data systems?
- What are the factors (e.g. pricing models, organizational usage, inefficiencies) that lead to such inflated costs?
- What are the signals that you collect for identifying targets for optimization and tuning?
- Can you describe how the Bluesky mission control platform is architected?
- What are the current areas of uncertainty or active research that you are focused on?
- What is the workflow for a team or organization that is adding Bluesky to their system?
- How does the usage of Bluesky change as teams move from the initial optimization and dramatic cost reduction into a steady state?
- What are the most interesting, innovative, or unexpected ways that you have seen teams approaching cost management in the absence of Bluesky?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Bluesky?
- When is Bluesky the wrong choice?
- What do you have planned for the future of Bluesky?
Contact Info
- Mingsheng
- @mingshenghong on Twitter
- Zheng
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
- Bluesky Data
- RocksDB
- Snowflake
- Trino
- Firebolt
- Bigquery
- Hive
- Vertica
- Michael Stonebraker
- Teradata
- C-Store Paper
- Ottertune
- dbt
- infracost
- Subtract: The Untapped Science of Less by Leidy Klotz
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
The AI-powered Podcast Player
Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
