How to Define Success Metrics for Data Products

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here. You can download their Data Mesh for Dummies e-book (info gated) here.

Samia's LinkedIn: https://www.linkedin.com/in/samia-r-b7b65216/

Khanh's LinkedIn: https://www.linkedin.com/in/khanhnchau/

Sheetal's Links:

Sheetal's LinkedIn: https://www.linkedin.com/in/sheetalpratik/

First Paper at ICEB Conference: https://easychair.org/publications/preprint/qZ3m

Interview in Harvard Business Review: https://hbr.org/resources/pdfs/comm/BeyondTechnology_CreatingBusinessValuewithDataMesh.pdf

Medium Blog on Saxo Bank Implementation: https://blog.datahubproject.io/enabling-data-discovery-in-a-data-mesh-the-saxo-journey-451b06969c8f

In this episode, guest host Samia Rahman, Director of Enterprise Data Strategy, Architecture, and Governance at life sciences company Seagen (guest of episode #67) facilitated a discussion with Sheetal Pratik, Director Engineering and leading India Data Integration Platform at Adidas (guest of episode #24), and Khanh Chau, Director of Cloud Data Architecture at Grainger (guest of episode #44). As per usual, all guests were only reflecting their own views.

The topic for this panel was a bit different with reflections and learnings from doing a second (or more) data mesh implementation from a practitioner standpoint - it's people who have the experience to reflect back on multiple implementations to give advice to their past selves and those earlier in their journeys after seeing multiple organizations implementing data mesh up close. Khanh Chau was the leading data architect for Northern Trust's data mesh implementation before taking up the same role at Grainger for their implementation. Sheetal was the head of data integration at Saxo Bank as part of their data mesh implementation before moving to Adidas where she is leading the India Data Integration Platforms and has also played integration lead for moving an on prem centralized integration platform to federated cloud integration platform. And Samia worked deeply on two implementations at Thoughtworks - including one closely with Zhamak - before starting at Seagen.

Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Scott's Top Takeaways:

1. Prepare to take many concepts from abstract to something concrete and explain it to many people. Repeatedly. And your definitions will change over time too. A big part of leading an implementation is about communication and keeping people on the same page and informed.
2. Similarly, prepare for confusion. People will go to different sources for information - including a lot of vendor content - to learn about data mesh. So keeping people aligned and understanding key aspects of data mesh are crucial. "What is data mesh?" can be a dangerous and difficult question when it shouldn't be.
3. It's important to learn from other organizations' implementations but your journey - if you are going for success and not simply copying - will be unique in what matters most when. It's easy to get overwhelmed trying to manage every aspect perfectly. So look to focus on learning and iterating along the way instead of getting it perfect upfront. Stepping back and looking at the 7 journeys across the panelists, they really don't look that similar.
4. When choosing which domain(s) to partner with, highest value or ROI can look like the most important metric but really, finding someone who is willing to take a risk and will also champion your solution when it succeeds is the most important. You do need to find use cases that have reasonable ROI but having a true partner is the most important part.
5. Data mesh isn't a solution to the business people - and it really shouldn't be to data people probably too. It's a way to address an ever growing challenge and issue within the organization. The business people feel the pain, talk to the pain. They don't care nearly as much about the how as you assume. Make them a tasty sausage, not give them a sausage factory tour.
6. Don't underestimate the fear of change and loss of control. Data mesh is a new approach and many are sick of new data approaches 😅 but people's fear of getting left behind or not being in control is a natural human reaction. Make sure people understand how this makes things better not just for the organization but also for them.
7. Defining data mesh success metrics - at the micro level of data products and at the macro level of the entire implementation - is still hard. There are a lot of measures you can use that are helpful - and do look into fitness functions - but it will take some effort to find good success metrics. And those metrics will change over time.
8. As some past guests have also noted, a good way to drive buy-in for those actually doing the data work is to make it faster to produce data for themselves, not just for others. If you can make getting access to data and creating something useful for the data producing domain quicker, many will lean in to leveraging the platform quickly.

Other Important Takeaways (many touch on similar points from different aspects):

1. Faster time to business outcomes is likely attractive to business partners to get buy-in AND a success metric you can actually somewhat reliably measure.
2. For success metrics for your implementation itself, look at time to deliver new products. Slow delivery might be because a domain isn't ready or leaning in but repeated slow delivery is probably a problem with the platform in some way.
3. It's tempting to try to start from the infrastructure aspects first but you have to focus on the value of doing something like data mesh. What is the value proposition for the organization? What is it for the specific person you are talking to?
4. The needs of your organization will look different than other organizations. You probably can't know exactly what will end up being your big focus ahead of time. A lot of the learnings and realizations can only happen when you're in the midst of your journey.
5. Because every part of an organization matures differently, each domain generally has developed a way of working that fits to their domain but makes it very hard to then standardize the ways of working or sharing data. You need to build central frameworks for collaborating but still allowing domains to work (mostly) in their existing ways. Otherwise, you are probably fighting a losing battle.
6. It's easy to lose sight of how important the business partners are to your data mesh success. They can provide a ton of leverage to driving buy-in, budget, etc. Or, they can provide a ton of headwinds/resistance. Find your good partners and make sure you're aligned on making your overall data mesh approach a success.
7. Similarly, at its core, data mesh is about business transformation. Yes, transformation driven by changing how we interact with data and do data work but business transformation nonetheless.
8. To get funding/sponsorship for something beyond the first few data products, you have to show the value proposition, especially in a soft economic environment. You aren't going to do that by talking technology. It's important to show a good return soon for sure but also a broader vision.
9. Technological solutions don't equate to business value to most business people. Talk to them about what they care about.
10. With every success, you can build momentum. But that momentum only builds if you're selling the successes, telling people and proving out the value. Data leaders need to be as much about showing off the results as achieving them unfortunately.
11. It can be effective to mesh buy-in to point out how decentralized efforts will probably happen - e.g. shadow IT - but they become their own little IT fiefdoms if not at least coordinated in some way through a central mechanism. It doesn't need to be ownership but if you don't have a central framework and ruleset, you end up with silos, spending more money on many platforms, and worse quality data.
12. As Khanh said regarding governance, "So how do we bring this whole ecosystem of technologies [together] in such a way that is safe for people to do things faster and meet all the regulatory compliance?" Two key words in there are safe and faster, don't make governance about gates, make it about getting to data production more quickly but still safely.
13. While data mesh may not itself be cheaper than doing nothing, it often results in cost savings relatively quickly by removing duplication of work. If everyone is at least broadcasting what they are working on, then there is far less overlap that might cause confusion and two teams doing essentially the same thing.
14. Trying to build the entire platform ahead of time is a really bad idea but you also can't come to the business with an inferior, clunky, hard-to-use, immature solution. Make sure your organization has some platform capabilities to not make it a huge hassle to build out early data products.
15. Don't try to onboard every domain at the start of your journey. Start to seed the conversation with them, talk to the pain points, but you can't handle 30 domains coming onboard at once.
16. When driving buy-in, don't only focus on executives. You need people throughout the organization leaning in. If you need new skills to accomplish data mesh, you need people who will acquire those skills and level up their data fluency. Talk to the people doing the actual day-to-day work.
17. You potentially don’t need a huge budget to start a data mesh journey. You can start lean - yes, you will have leaner results but if you can prove faster time to business outcomes, you can generate more and more momentum.
18. Build momentum around your mindset shift. You aren't going to convince people overnight.
19. It's important for domains to take over ownership of data but that also doesn't happen overnight. And your initial ownership will probably not be as strong as you'd like. And it probably won't involve the operational plane system experts/developers as much as you'd like. But that will evolve and get better. You have to get started with okay, not perfect. Crawl, walk, run.
20. You probably need to apply DevOps principles to your data products where the data product developers also support the data products, do the operations aspect of them. But it might be hard to get people really bought in on that when your platform is still pretty nascent.
21. You want to build feedback mechanisms around data products to drive domains to make them better. But there are many concepts around data product feedback being touted by vendors but few people are happy with what they've built or are getting from vendors. The best feedback mechanism is probably a culture shift towards providing more/better feedback.
22. As Samia said, "It's not really on us as governors to govern and … force it on you but to empower you so that you are doing the right things for yourself and for your organization."
23. There's a ton of interest in how generative AI can play in data mesh but it's still far too early for it to have a big impact.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf