Data Engineering Podcast
Surveying The Market Of Database Products
Oct 30, 2023
Tanya Bragin, an experienced product manager for major vendors, shares insights on how to approach tool selection in the database market. Topics include open-source technologies, trends in database market, challenges in data projects, importance of data observability tools, and future trends.
47:12
Episode guests
AI Summary
AI Chapters
Episode notes
Podcast summary created with Snipd AI
Quick takeaways
- OLAP databases will be incorporated from the beginning of building applications, alongside OLTP databases.
- Unbundling cloud data warehouses and leveraging data lakes will continue to gain traction.
- Advancements in marketing and sales are anticipated through the blend of vector stores, gen AI, and domain-specific knowledge.
Deep dives
OLAP Continues to Grow in Prominence
The prediction is that OLAP (Online Analytical Processing) will continue to gain importance and developers will start incorporating OLAP databases from the beginning of building their applications, alongside OLTP (Online Transactional Processing) databases. This trend is driven by the increasing investment and awareness in real-time analytics and the need for interactive data-driven applications.
Unbundling of Cloud Data Warehouses
There is a predicted trend of unbundling cloud data warehouses as data lakes rise in prominence. The concept of data lakes, where different open technologies can be leveraged on a shared object store, makes sense for various use cases. While specialized storage may still be preferred for certain workloads, the idea of having a single place to store data and allow multiple analytical engines to access it will continue to gain traction.
The Impact of Vector Stores and Gen AI
The hype around vector stores and gen AI is expected to lead to new applications and innovative solutions. While the exact impact is unclear, it is believed that a blend of approaches, incorporating large language models and domain-specific knowledge, will be the way forward. These developments are anticipated to drive advancements in marketing, sales, and various other industries.
Challenges in Adopting Analytical Databases
One of the challenges in adopting analytical databases, as seen with ClickHouse, is the need to understand and optimize data organization and semantics. Flattening data and transforming it into a suitable format for querying is crucial to fully utilize the capabilities of an analytical database. Education around these nuances is essential to ensure successful adoption.
The Importance of Data Observability
Data observability tools are becoming increasingly important in the database market. Database vendors like ClickHouse focus on providing good self-observability, ensuring that information about the database and its changes is easily accessible. Integration with data observability tools allows users to monitor, trace, and profile their databases, enabling confident and efficient operation.
2.
The Journey in the Data Space: From Custom Databases to Open Source Technologies
13min
5.
Challenges in Data Projects, Collaboration, and Unbundling Cloud Data Warehouses
4min
6.
Exploring the Importance of Data Observability Tools and Database Engine Selection
11min
Summary
Databases are the core of most applications, whether transactional or analytical. In recent years the selection of database products has exploded, making the critical decision of which engine(s) to use even more difficult. In this episode Tanya Bragin shares her experiences as a product manager for two major vendors and the lessons that she has learned about how teams should approach the process of tool selection.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack
- You shouldn't have to throw away the database to build with fast-changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. Whether it’s real-time dashboarding and analytics, personalization and segmentation or automation and alerting, Materialize gives you the ability to work with fresh, correct, and scalable results — all in a familiar SQL interface. Go to dataengineeringpodcast.com/materialize today to get 2 weeks free!
- This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold
- Data projects are notoriously complex. With multiple stakeholders to manage across varying backgrounds and toolchains even simple reports can become unwieldy to maintain. Miro is your single pane of glass where everyone can discover, track, and collaborate on your organization's data. I especially like the ability to combine your technical diagrams with data documentation and dependency mapping, allowing your data engineers and data consumers to communicate seamlessly about your projects. Find simplicity in your most complex projects with Miro. Your first three Miro boards are free when you sign up today at dataengineeringpodcast.com/miro. That’s three free boards at dataengineeringpodcast.com/miro.
- Your host is Tobias Macey and today I'm interviewing Tanya Bragin about her views on the database products market
Interview
- Introduction
- How did you get involved in the area of data management?
- What are the aspects of the database market that keep you interested as a VP of product?
- How have your experiences at Elastic informed your current work at Clickhouse?
- What are the main product categories for databases today?
- What are the industry trends that have the most impact on the development and growth of different product categories?
- Which categories do you see growing the fastest?
- When a team is selecting a database technology for a given task, what are the types of questions that they should be asking?
- Transactional engines like Postgres, SQL Server, Oracle, etc. were long used as analytical databases as well. What is driving the broad adoption of columnar stores as a separate environment from transactional systems?
- What are the inefficiencies/complexities that this introduces?
- How can the database engine used for analytical systems work more closely with the transactional systems?
- When building analytical systems there are numerous moving parts with intricate dependencies. What is the role of the database in simplifying observability of these applications?
- What are the most interesting, innovative, or unexpected ways that you have seen Clickhouse used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on database products?
- What are your prodictions for the future of the database market?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
- Clickhouse
- Elastic
- OLAP
- OLTP
- Graph Database
- Vector Database
- Trino
- Presto
- Foreign data wrapper
- dbt
- OpenTelemetry
- Iceberg
- Parquet
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Sponsored By:
- Miro:  Data projects are notoriously complex. With multiple stakeholders to manage across varying backgrounds and toolchains even simple reports can become unwieldy to maintain. Miro is your single pane of glass where everyone can discover, track, and collaborate on your organization's data. I especially like the ability to combine your technical diagrams with data documentation and dependency mapping, allowing your data engineers and data consumers to communicate seamlessly about your projects. Find simplicity in your most complex projects with Miro. Your first three Miro boards are free when you sign up today at [dataengineeringpodcast.com/miro](https://www.dataengineeringpodcast.com/miro).
- Rudderstack:  Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack)
- Materialize:  You shouldn't have to throw away the database to build with fast-changing data. Keep the familiar SQL, keep the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. That is Materialize, the only true SQL streaming database built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI. Built on Timely Dataflow and Differential Dataflow, open source frameworks created by cofounder Frank McSherry at Microsoft Research, Materialize is trusted by data and engineering teams at Ramp, Pluralsight, Onward and more to build real-time data products without the cost, complexity, and development time of stream processing. Go to [materialize.com](https://materialize.com/register/?utm_source=depodcast&utm_medium=paid&utm_campaign=early-access) today and get 2 weeks free!
- Datafold:  This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting [dataengineeringpodcast.com/datafold](https://www.dataengineeringpodcast.com/datafold) today!
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
