The Data Generalist's Vision Quest (LIVE w/ Stephen Bailey)
Dec 2, 2022
auto_awesome
Stephen Bailey, data engineer at Whatnot and writer of an incredibly entertaining data substack, discusses the challenges of being a generalist in the data field. They explore the supportive dbt community and the importance of collaboration. They also discuss gaining unique perspectives, finding creative expression, and their hopes for the future of the data community.
Being an effective generalist in analytics engineering is crucial for connecting different parts of the business and bridging the gap between analysts and machine learning engineers.
The role of data engineers has evolved from specialized big data engineers to life cycle data engineers who focus on ensuring reliable and trustworthy data flows to meet the distinct requirements of different stakeholders across the organization.
Deep dives
The Value of Being a Generalist in Analytics Engineering
In this podcast episode, Stephen Bailey, a data engineer at Whatnot, discusses the importance of being an effective generalist in the field of analytics engineering. He emphasizes the need to be a Swiss Army Knife for organizations, possessing a wide range of skills and the ability to adapt to various roles and responsibilities. Bailey highlights the value of generalists who can connect different parts of the business, understand diverse stakeholder requirements, and effectively bridge the gap between analysts and machine learning engineers. He believes that generalists bring a unique perspective and are crucial in driving the organization's adaptability and speed of operation.
The Evolution of Data Teams and the Changing Role of Data Engineers
Bailey muses on the changing landscape of data teams over the years. He suggests that 10 years ago, there may not have been distinct data teams, as individuals in the organization used accessible tools to source their own data and solve problems. With the advent of big data, the need for data teams arose to manage and curate large-scale data systems. Bailey describes the shift from specialized big data engineers to life cycle data engineers who focus on the stewardship and curation of data throughout its ingestion, transformation, and delivery cycles. He emphasizes that data engineers now have a more business-oriented role, ensuring reliable and trustworthy data flows to meet the distinct requirements of different stakeholders across the organization.
The Value of Creativity and Experimentation in the Data Community
The podcast delves into the creative realm of writing and communication within the data community. Bailey explores the importance of creativity and experimentation in expressing ideas and conveying complex concepts. He shares his own experiences with creative writing and humor to communicate innovative ideas in a distinct and engaging manner. Bailey emphasizes the need for data professionals to be adaptable, experimental, and willing to explore non-linear solutions to problems in order to drive progress and effectively meet the evolving needs of organizations and the data ecosystem at large.
Stephen Bailey, data engineer at Whatnot and writer of an incredibly entertaining data substack, joins Tristan for a follow-up conversation to Stephen’s Coalesce talk, “Excel at nothing: how to be an effective generalist.”