Building And Managing Data Teams And Data Platforms In Large Organizations With Ashish Mrig

Jan 23, 2022

Ashish Mrig, head of the data analytics platform at Wayfair and organizer of a local data engineering meetup, shares his insights into managing data teams in large organizations. He discusses the challenges of balancing stakeholder demands with technological advancement, particularly in cloud migration. Ashish delves into the evolution of data engineering roles, from analytics to machine learning workloads, and emphasizes the importance of strategic planning in data architecture. He also touches on common pitfalls in data management and the future of data quality and technology.

Ask episode

Chapters

Transcript

Episode notes

Intro

00:00 • 3min

Journey Through the Data Ecosystem: From Technician to Manager

02:54 • 2min

Navigating Data Engineering at Scale

04:36 • 27min

Evolving Data Engineering: Balancing Analytics and Machine Learning Workloads

31:14 • 6min

Navigating Data Management Challenges

37:00 • 8min

Navigating Analytics Misconceptions and Data Architecture Challenges

44:37 • 3min

Exploring Innovations in Data Engineering and Quality Management

47:43 • 3min

Evolution of Data Management Technology and Future Challenges

50:33 • 2min

Summary

Data engineering is a relatively young and rapidly expanding field, with practitioners having a wide array of experiences as they navigate their careers. Ashish Mrig currently leads the data analytics platform for Wayfair, as well as running a local data engineering meetup. In this episode he shares his career journey, the challenges related to management of data professionals, and the platform design that he and his team have built to power analytics at a large company. He also provides some excellent insights into the factors that play into the build vs. buy decision at different organizational sizes.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud. Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow. Now all the data users can use software engineering best practices – git, tests and continuous deployment with a simple to use visual designer. How does it work? – You visually design the pipelines, and Prophecy generates clean Spark code with tests on git; then you visually schedule these pipelines on Airflow. You can observe your pipelines with built in metadata search and column level lineage. Finally, if you have existing workflows in AbInitio, Informatica or other ETL formats that you want to move to the cloud, you can import them automatically into Prophecy making them run productively on Spark. Create your free account today at dataengineeringpodcast.com/prophecy.
The only thing worse than having bad data is not knowing that you have it. With Bigeye’s data observability platform, if there is an issue with your data or data pipelines you’ll know right away and can get it fixed before the business is impacted. Bigeye let’s data teams measure, improve, and communicate the quality of your data to company stakeholders. With complete API access, a user-friendly interface, and automated yet flexible alerting, you’ve got everything you need to establish and maintain trust in your data. Go to dataengineeringpodcast.com/bigeye today to sign up and start trusting your analyses.
Your host is Tobias Macey and today I’m interviewing Ashish Mrig about his path as a data engineer

Interview

Introduction
How did you get involved in the area of data management?
You currently lead a data engineering team at a relatively large company. What are the topics that account for the majority of your time and energy?
What are some of the most valuable lessons that you’ve learned about managing and motivating teams of data professionals?
What has been your most consistent challenge across the different generations of the data ecosystem?
How is your current data platform architected?
Given the current state of the technology and services landscape, how would you approach the design and implementation of a greenfield rebuild of your platform?
What are some of the pitfalls that you have seen data teams encounter most frequently?
You are running a data engineering meetup for your local community in the Boston area. What have been some of the recurring themes that are discussed in those events?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Support Data Engineering Podcast