Join Benjamin Rogojan, a seasoned data engineering consultant, along with Gable's Chad Sanderson, NAO's CTO Christophe Blefari, and Acryl Data's Maggie Hays for a lively discussion on the ever-evolving world of data. They delve into their diverse career journeys and transformative experiences in data engineering. Expect insights on the rise of modern data tools, personal anecdotes about creating data lakes, and the emerging challenges in data governance. Discover how shared ownership of data models enhances collaboration and drives business value!
Read more
AI Summary
AI Chapters
Episode notes
auto_awesome
Podcast summary created with Snipd AI
Quick takeaways
The evolution of data engineering has shifted from traditional tools to modern technologies, emphasizing flexibility and efficiency in data processing.
Establishing a culture of data governance and treating data as a product are crucial for organizations to enhance data quality and derive business value.
Deep dives
Evolution of Data Engineering Roles
Data engineering roles have experienced significant transformations over the years as the field has evolved. Initially, the focus was on traditional tools like SAS and SQL, which served organizations in handling data with limited resources and capabilities. As technology progressed, data engineers began adopting newer tools and methodologies, such as Python and cloud-based solutions, that allowed for more flexible and efficient data processing. This evolution highlights that data engineering is now crucial for organizations looking to leverage their data assets effectively, adapting to changing tools and practices in the tech landscape.
Challenges of Governance and Compliance
The rapid advancement of data tools and technologies has outpaced the development of effective governance and compliance practices. Organizations often struggle with ensuring their data assets are well-documented and understood, especially in a landscape where new data resources can be easily created. This has led to an increase in poorly managed data pipelines, risking data quality and complicating compliance tasks. Therefore, fostering a culture of data governance is essential in order to keep pace with the ever-growing data ecosystem.
The Need for Data Model Maintenance
Data models serve as the backbone of analytics, yet their maintenance often takes a backseat in data-driven organizations. Without a systematic approach to reviewing and updating these models, teams can end up with countless unused or inefficient models that contribute to data entropy. Implementing practices such as periodic reviews of data models can help manage this complexity and ensure relevance. Additionally, establishing a formalized deprecation strategy could further streamline these efforts, preventing a build-up of unused resources.
Data as a Product
Treating data as a product is becoming increasingly critical for organizations aiming to maximize the value derived from their data assets. This paradigm shift encourages a focus on understanding how data supports business goals and driving user engagement, similar to how software features are developed and monitored for effectiveness. Building out this approach involves establishing clear metrics for evaluating the value users derive from data models, akin to how software products are continually assessed for user interaction and utility. Embracing this mindset can guide organizations in refining their data processes and ensuring alignment with overall strategic objectives.
//Abstract
If there is one thing that is true, it is data is constantly changing. How can we keep up with these changes? How can we make sure that every stakeholder has visibility? How can we create a culture of understanding around data change management?
//Bio
- Benjamin Rogojan: Data Science And Engineering Consultant @ Seattle Data Guy
- Chad Sanderson: CEO & Co-Founder @ Gable
- Christophe Blefari: CTO & Co-founder @ NAO
- Maggie Hays: Founding Community Product Manager, DataHub @ Acryl Data
A big thank you to our Premium Sponsors @Databricks, @tecton8241, & @onehouseHQfor their generous support!
Get the Snipd podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share & Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode