Data Brew by Databricks

In this session, we discuss the lessons learned with Lara Minor, Senior Enterprise Data Manager at Columbia Sportswear, on how her team achieved a 70% reduction in pipeline creation time. This had reduced ETL workload times from four hours with previous data warehouses to minutes enabling near real-time analytics. Her team migrated from multiple legacy data warehouses, run by individual lines of business, to a single scalable, reliable, performant data lake.See more at databricks.com/data-brew

In this podcast, Michael Armbrust, the creator of Spark SQL, discusses the conception and evolution of Delta Lake, efficient querying and troubleshooting slow queries, optimizing performance and query speed, understanding partitioning and Z Order, and exciting features for data ingestion and schema handling in Delta Lake.

The podcast discusses the transition to big data, cluster management, and the democratization of data. It explores the challenges of data lakes and introduces Delta Lake as a solution. The concept of a lakehouse paradigm is explored, enabling data-driven enterprises. The podcast also provides advice for building and leveraging lake houses for data science and machine learning.

Barry Devlin, Susan O'Connell, and Donald Farmer discuss the evolution from data warehousing to data lakes. They highlight the importance of data warehousing in the 90s, benefits of data lakes, challenges of working without management and governance, integration and collaboration challenges, resolving paradoxes for data lake houses, and the significance of information management and people aspects in dealing with data.