Data Engineering Podcast Building A Data Mesh Platform At PayPal
46:54
Data Mesh at PayPal
- Jean-Georges Perrin leads teams building PayPal's new data platform, structured as a data mesh.
- His background includes consulting, entrepreneurship, and 15 years of data platform design.
Organic Data Mesh
- PayPal's data mesh implementation wasn't initially planned as such but organically aligned with the concept.
- Recognizing this alignment simplified the project and enhanced the data platform's vision.
Data Product Value
- Focus on delivering value as the core principle for defining a data product.
- Start with a small, valuable unit (data quantum), which can later be integrated into a larger data mesh.
Get the Snipd Podcast app to discover more snips from this episode
Get the app 1 chevron_right 2 chevron_right 3 chevron_right 4 chevron_right 5 chevron_right 6 chevron_right 7 chevron_right 8 chevron_right 9 chevron_right 10 chevron_right 11 chevron_right 12 chevron_right 13 chevron_right 14 chevron_right 15 chevron_right 16 chevron_right 17 chevron_right 18 chevron_right
Introduction
00:00 • 2min
Data Mesh - I Hate It
01:59 • 2min
Data Mesh at PayPal
03:47 • 3min
Is the Data Mesh Ever Done?
07:14 • 2min
Is There a Minimum Granularity for a Data Mesh?
09:20 • 3min
Data Meshes to Decentralize Data Product Owners
12:01 • 2min
What's the Difference Between a Product Organization and a Software Organization?
14:20 • 3min
Data Contracts
17:10 • 4min
Is It Better Than Nothing?
21:21 • 4min
Data Mesh Tooling - The Sidecar Approach
24:54 • 3min
The Challenges of Building a Data Mesh
27:35 • 3min
Data Mesh - What's the Difference?
30:33 • 2min
Data Product Life Cycle - What's the Biggest Surprise?
32:46 • 2min
Data Mesh - What Are Some of the Most Valuable Lessons You've Learned?
34:33 • 2min
Data Mesh Integration - What Are the Foundational Principles?
36:16 • 4min
Jamaox Books, Data Mesh Radio
39:47 • 3min
Data Mesh at PayPal - Lessons Learned
43:14 • 2min
The Biggest Gap in Data Governance
44:51 • 2min
Summary
There has been a lot of discussion about the practical application of data mesh and how to implement it in an organization. Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. In this episode he shares that journey and the combination of technical and organizational challenges that he encountered in the process.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Are you tired of dealing with the headache that is the 'Modern Data Stack'? We feel your pain. It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. It ends up being anything but that. Setting it up, integrating it, maintaining it—it’s all kind of a nightmare. And let's not even get started on all the extra tools you have to buy to get it to do its thing. But don't worry, there is a better way. TimeXtender takes a holistic approach to data integration that focuses on agility rather than fragmentation. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. If you're fed up with the 'Modern Data Stack', give TimeXtender a try. Head over to dataengineeringpodcast.com/timextender where you can do two things: watch us build a data estate in 15 minutes and start for free today.
- Your host is Tobias Macey and today I'm interviewing Jean-Georges Perrin about his work at PayPal to implement a data mesh and the role of data contracts in making it work
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you start by describing the goals and scope of your work at PayPal to implement a data mesh?
- What are the core problems that you were addressing with this project?
- Is a data mesh ever "done"?
- What was your experience engaging at the organizational level to identify the granularity and ownership of the data products that were needed in the initial iteration?
- What was the impact of leading multiple teams on the design of how to implement communication/contracts throughout the mesh?
- What are the technical systems that you are relying on to power the different data domains?
- What is your philosophy on enforcing uniformity in technical systems vs. relying on interface definitions as the unit of consistency?
- What are the biggest challenges (technical and procedural) that you have encountered during your implementation?
- How are you managing visibility/auditability across the different data domains? (e.g. observability, data quality, etc.)
- What are the most interesting, innovative, or unexpected ways that you have seen PayPal's data mesh used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on data mesh?
- When is a data mesh the wrong choice?
- What do you have planned for the future of your data mesh at PayPal?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
- Data Mesh
- O'Reilly Book (affiliate link)
- The next generation of Data Platforms is the Data Mesh
- PayPal
- Conway's Law
- Data Mesh For All Ages - US, Data Mesh For All Ages - UK
- Data Mesh Radio
- Data Mesh Community
- Data Mesh In Action
- Great Expectations
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Sponsored By:
- TimeXtender:  TimeXtender is a holistic, metadata-driven solution for data integration, optimized for agility. TimeXtender provides all the features you need to build a future-proof infrastructure for ingesting, transforming, modelling, and delivering clean, reliable data in the fastest, most efficient way possible. You can't optimize for everything all at once. That's why we take a holistic approach to data integration that optimises for agility instead of fragmentation. By unifying each layer of the data stack, TimeXtender empowers you to build data solutions 10x faster while reducing costs by 70%-80%. We do this for one simple reason: because time matters. Go to [dataengineeringpodcast.com/timextender](https://www.dataengineeringpodcast.com/timextender) today to get started for free!