Data Mesh Radio #68 The Build vs. Buy Dilemma of Data Platforms - Interview w/ Doron Porat
Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/
Please Rate and Review us on your podcast app of choice!
If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here
Episode list and links to all available episode transcripts here.
Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.
Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here
Doron's LinkedIn: https://www.linkedin.com/in/porat-doron/
Our journey towards an open data platform: https://medium.com/yotpoengineering/our-journey-towards-an-open-data-platform-8cfac98ef9f5
A simplified, lightweight ETL Framework based on Apache Spark by Yotpo: https://github.com/YotpoLtd/metorikku
The Data Swamp (in Hebrew): https://open.spotify.com/show/5YDdtRhp1RVw7r5fbYFtPQ?si=5x4HzOyhTX6n46HqY5kV6w&nd=1
In this episode, Scott interviewed Doron Porat, a Data Infrastructure Leader at the SaaS company Yotpo.
Some crucial points Doron made:
1) Be kind to yourself when you make mistakes - it's worse to stagnate so don't be afraid of change and making choices
2) Build versus buy is always tough but don't let your ego get in the way and push you towards building everything
3) If you do buy, build a close relationship with your vendors to help influence the roadmap and have an outlet if you are having issues
4) A data platform team's job is to drive usage as usage means creating value - drive towards that and set your KPIs around platform usage
5) There will likely be many different types of consumers of your data platform - work to improve / optimize the user experience for most folks
Doron is a technologist at heart so for each decision she instinctually wants to build instead of buy. And at the start of building out the data platform for Yotpo, that was typically her decision. But as the demands for more and more capabilities from the platform, the increasing ubiquity and quality/scalability of as-a-service offerings, and the growing need to drive usage and developer happiness instead of manage cool tech, she started to consume more and more managed services.
When you are building out the platform, vendors, in the long run, can often better serve your needs because they have a whole lot of people focused specifically on making what you use better. You need to make bets on the vendors getting to where you need them to be and sometimes they don't pay off either. To up the chance of making those bets pay off, Doron recommends building relationships with your vendors to influence their roadmaps and get help when necessary.
Doron strongly recommends putting together a framework for evaluating build/buy decisions. Some of the factors she considers are how extensible is the offering, is this taking on too many challenges in one solution, cost, the cost of later migration to or from a managed service, open source compatibility, etc. One thing Doron talked about that many teams seem to struggle with is the ego hit of saying someone else managing a service we use will drive more value. That's always tough but needs to be addressed.
Doron talked about the strong need to drive your platform forward, not just be responsive. Provide a roadmap, set time aside for innovation, etc.
Doron made the point there is a difference between learning to leverage a tool and operate it. You want to make sure you build out the knowledge around leveraging a tool whether you are operating it yourself or not. It's an interesting balance when you build that you don't focus too much on one or the other.
Doron talked about the general job of a data platform team is to drive usage because that usage means creating value. Serving critical needs for users is crucial to driving adoption and focus the user experience on the business logic - no matter how cool the tech is, a data platform team's job isn't to expose that to users. And your team's KPIs should reflect usage. However, there are all kinds of users - on both the consumer and producer side - that you need to serve so don't create a one-size-fits-all experience.
If your data platform isn't easy to adopt, no one will want to use it. Your data platform should make things fun and casual to do. There needs to be a big focus on enabling data testing including staging environments for data - that will drive usage.
Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/
If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/
If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here
All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf