An Exploration Of The Open Data Lakehouse And Dremio's Contribution To The Ecosystem
Data Engineering Podcast
00:00
Dremio - What's the Most Unexpected Way to Leverage Spark to Do Parallel Reads?
One of our customers actually leverages it as a single node and runs it in each of their little IoT edge platform so they can leverage Dremio. So to me that was probably the most expected and they've actually been doing that for a bit now. I think one of the most innovative and this is more off the top of my head but one of our customers is actually leveraging Spark to do parallel reads from Dremio Leveraging arrow flight to read that data in very quicklyso they can actually do machine learning on core business data.
Transcript
Play full episode
Transcript
Episode notes
Summary
The "data lakehouse" architecture balances the scalability and flexibility of data lakes with the ease of use and transaction support of data warehouses. Dremio is one of the companies leading the development of products and services that support the open lakehouse. In this episode Jason Hughes explains what it means for a lakehouse to be "open" and describes the different components that the Dremio team build and contribute to.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
- You wake up to a Slack message from your CEO, who’s upset because the company’s revenue dashboard is broken. You’re told to fix it before this morning’s board meeting, which is just minutes away. Enter Metaplane, the industry’s only self-serve data observability tool. In just a few clicks, you identify the issue’s root cause, conduct an impact analysis—and save the day. Data leaders at Imperfect Foods, Drift, and Vendr love Metaplane because it helps them catch, investigate, and fix data quality issues before their stakeholders ever notice they exist. Setup takes 30 minutes. You can literally get up and running with Metaplane by the end of this podcast. Sign up for a free-forever plan at dataengineeringpodcast.com/metaplane, or try out their most advanced features with a 14-day free trial. Mention the podcast to get a free "In Data We Trust World Tour" t-shirt.
- RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
- Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
- Your host is Tobias Macey and today I’m interviewing Jason Hughes about the work that Dremio is doing to support the open lakehouse
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Dremio is and the story behind it?
- What are some of the notable changes in the Dremio product and related ecosystem over the past ~4 years?
- How has the advent of the lakehouse paradigm influenced the product direction?
- What are the main benefits that a lakehouse design offers to a data platform?
- What are some of the architectural patterns that are only possible with a lakehouse?
- What is the distinction you make between a lakehouse and an open lakehouse?
- What are some of the unique features that Dremio offers for lakehouse implementations?
- What are some of the investments that Dremio has made to the broader open source/open lakehouse ecosystem?
- How are those projects/investments being used in the commercial offering?
- What is the purchase/usage model that customers expect for lakehouse implementations?
- How have those expectations shifted since the first iterations of Dremio?
- Dremio has its ancestry in the Drill project. How has that history influenced the capabilities (e.g. integrations, scalability, deployment models, etc.) and evolution of Dremio compared to systems like Trino/Presto and Spark SQL?
- What are the most interesting, innovative, or unexpected ways that you have seen Dremio used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Dremio?
- When is Dremio the wrong choice?
- What do you have planned for the future of Dremio?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
- Dremio
- Dremio Sonar
- Dremio Arctic
- DML == Data Modification Language
- Spark
- Data Lake
- Trino
- Presto
- Dremio Data Reflections
- Tableau
- Delta Lake
- Apache Impala
- Apache Arrow
- DuckDB
- Google BigLake
- Project Nessie
- Apache Iceberg
- Hive Metastore
- AWS Glue Catalog
- Dremel
- Apache Drill
- Arrow Gandiva
- dbt
- Airbyte
- Singer
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
The AI-powered Podcast Player
Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
