The Challenges of Scaling a Data Platform

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts (most interviews from #32 on) here

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here. You can download their Data Mesh for Dummies e-book (info gated) here.

Kim's LinkedIn: https://www.linkedin.com/in/vtkthies/

Mike's LinkedIn: https://www.linkedin.com/in/2mikealvarez/

Ferd's LinkedIn: https://www.linkedin.com/in/ferdscheepers/

Omar's LinkedIn: https://www.linkedin.com/in/kmaomar/

In this episode, guest host Kim Thies, Director of Intelligence Automation at PayPal facilitated a discussion with Ferd Scheepers, Chief Information Architect at ING, Mike Alvarez, Former VP of Digital Services at a large healthcare distribution company (guest of episode #236), and Omar Khawaja, Head of Business Intelligence at Roche (guest of episode #96). As per usual, all guests were only reflecting their own views.

Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Before we jump in, I think the main takeaway here would be a data mesh implementation leader's journey can be a lonely one. Find peers and exchange information. You can reach out to me (Scott) but there are also many leaders that want to exchange information with each other. The other is the meaning of journey: it's never done; be prepared to continue to push - it can feel Sisyphean but it's important to keep moving forward and expect to continue to drive buy-in.

Scott's Top Takeaways:

1. Everyone sees the 'Instagram photos' version of other organizations' data mesh journeys - it's not the reality. Everyone is struggling with certain aspects of data mesh because if this were easy, people would read Zhamak's book and be done with it. It's just not realistic to expect that, give your leaders (yourself?) a break :)
2. It's incredibly important to understand you will get things 'wrong'. Scratch that, you will get MANY things wrong. But 'wrong' in data doesn't have to mean wrong for good/all time. It's about trying, learning, and iterating to better. Really look into fast fail practices. If you feel the need to get everything right, data mesh is definitely not right for you right now.
3. A champion early adopter is crucial to drive a data mesh implementation. There needs to be someone who will partner with you that is "brave enough to try new things" as Omar said.
4. It's easy to focus on so many aspects of data mesh and forget that the mindset shift and cultural change are the biggest challenges for most organizations. They are the most squishy/least tangible but that doesn't mean they aren't absolutely crucial.
5. Teams need the autonomy and empowerment to move at the right speed for them. The more friction in their data work, the more time - and effort - it takes to deliver value and the world can have moved on. You have to give teams the capability to strike while the iron's hot. That's a good way to drive buy-in: we're going to give you the ability to move at the speed of business.
6. Be very clear on expectations. What do we owe each other? Focus on how data mesh drives value for them but also how it drives value for the organization. What is the output of their work, what value comes from what they did or will do? It will be quite difficult to change this kind of thinking overnight.
7. Similarly, be very clear on responsibilities and be very clear on target outcomes. It's easy to get lost in the work instead of what are we trying to achieve.
8. A good success story doesn't have to be a massive win. Time to delivery is really valuable so showing getting something into MVP in a few weeks will get a number of teams excited. Driving that lower is of significant value - and it's a tangible value for many business partners where they can reasonably share about the impact to their business.

Other Important Takeaways (many touch on similar points from different aspects):

1. There's a very good reason why we refer to them as data mesh journeys :) You can't treat this as a project. Think of your physical fitness and health. It's an ongoing journey with plenty of trials and tribulations.
2. It's okay if data mesh is not a fit for your organization. There's no shame in that. Be honest when considering if it's a fit or if you just want to do it because it sounds like a good solution in the abstract.
3. In general, look to collect reasons for past failures prior to doing data mesh. Not to throw others or past ways under the bus but to emphasize that the current ways aren't working as well as we might like. Proof points really help convince others.
4. It's always a challenge to maintain autonomy and empowerment without it leading to silos. There isn't a magic formula. It's something you will have to constantly keep a watch on but you're not the only one.
5. There is a balance between autonomy and interoperability. You'll probably get it 'not great' at first and THAT'S OKAY. Work to find the right balance - or at least an acceptable balance - as you move forward. And that balance will likely shift.
6. The focus of the platform should be on making users' lives simpler so they can actually own the data. It's easy to fall into focusing on the tech but the value is in friction reduction for users and the abstractions you offer them.
7. Be prepared to need to continually drive buy-in. Gravity around historical - legacy? - data management practices is STRONG. You will have people wanting to use an enterprise data warehouse or a data lake setup or not wanting to focus on integration with the rest of the organization. Announcing you are doing data mesh doesn't magically mean everyone is bought in.
8. Similarly, be prepared to explain concepts and terms repeatedly. Data mesh - and data work in general - is not the main focus of most people in your organization. Do your best to have the patience to work with people to drive towards a better understanding.
9. Don't focus overly on terminology - see my unicorn farts theory… - focus on what the change in work means for people and what outcomes you are trying to achieve. Business leaders and users don't care that you are doing data mesh - data mesh is a shared language for the data people working on the implementation. Especially don't try to explicitly define federated computational governance :D
10. As Ferd said, look to "bring governance back to the essence of what we want to do." Don't get overly focused on governing everything instead of the right things. That can mean not taking on really difficult challenges right at the start of your journey by passing on use cases. That's okay, think about your capabilities and honestly assess them. Don't bite off more than you can chew.
11. Incremental progress is the best approach to most (all?) aspects of data mesh rather than trying to take giant leaps.
12. Early in your journey, likely focus more on quick wins that will give you some good internal marketing material - market those successes to build desire for additional domains to join, build the momentum of success. Also, think about limiting chances for failure early. Fast fail in general is crucial but you don't want a full 'fast fail' on your first use case.
13. Many things will not work or go great in a data mesh journey. That's the reality of being on the bleeding edge, being an explorer. It's very important to reflect not just on wins but take the learnings from what didn't work. If you don't reflect on your mistakes, you're far more likely to repeat them.
14. Your early champions and business partners can come from unexpected corners of the business. Look for bravery and a well-defined - and valuable - use case with limited initial scope. Be open to the idea of surprising initial use cases and working with unexpected lines of business/domains.
15. Try to directly tie business value generation to your data mesh work. Understand the value drivers for the organization and prioritize the work based on those value drivers.
16. To get buy-in, you don't have to offer domains the perfect solution, just one that is a better option than whatever else is available internally for their use case.
17. Another good potential hook for potential business partners is that data mesh - specifically handling your data as a product - has less technical debt so it can quickly pay for itself in saved time especially for use cases that are mandatory like regulatory compliance.
18. Be very diligent in putting teams together, especially early - throwing bodies at the problem won't solve anything but not having the right capabilities in the teams sets those teams up for failure. I recommend reading Zhamak's book on this topic re necessary capabilities in teams.
19. Incentivization to become a data producer can be quite hard but it's extremely important to get right. How can we think of value creation through data without the perception of monetization? Because that can feel like you are selling data externally.
20. For your first use case, you need to have the business and technology/data team working as one team - one _actual_ team, not just a team in name. That doesn't mean a huge re-org but you need people constantly sharing context and partnering to drive to value. Otherwise, you have a high risk of falling back to the same old ways of doing things.
21. Make failure not a catastrophe. As Vanya Seth says, "contain your blast radius."
22. It's crucial for people to understand that doing decentralized data means a lot of changes. Get people comfortable - or at least acutely aware - that we'll need new ways of working and delivering value.
23. It's probably easier in most organizations to get funding/buy-in if you focus deeply on the business use case. Yes, building data mesh capabilities has an overhead to the initial use case or two as you spend time and effort learning how to do data mesh and it's okay to talk about that. But if you aren't talking direct, tangible business value, getting buy-in - and more importantly funding - will likely be harder.
24. Make sure you have buy-in from consumers on use cases. If they aren't bought in, will they even consume? If you don't have users, the best product in the world is useless.
25. Budgetary issues - why would one domain create value for another if they aren't getting compensated for it? - are extremely common. There isn't an easy answer here unfortunately.
26. As stated above, in general, it's often lonely being the person leading a data mesh journey. Who do you talk to? There's me (Scott) but find other leaders and network.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf