#233 Panel: A Head Data Architect's View of Data Mesh - Led by Khanh Chau w/ Balvinder Khurana, Yushin Son, and Carlos Saona

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here. You can download their Data Mesh for Dummies e-book (info gated) here.

#233 Panel: A Head Data Architect's View of Data Mesh - Led by Khanh Chau w/ Balvinder Khurana, Yushin Son, and Carlos Saona

Khanh's LinkedIn: https://www.linkedin.com/in/khanhnchau/

Balvinder's LinkedIn: https://www.linkedin.com/in/balvinder-khurana/

Yushin's LinkedIn: https://www.linkedin.com/in/yushin-son-30362b1/

Carlos' LinkedIn: https://www.linkedin.com/in/carlos-saona-vazquez/

In this episode, guest host Khanh Chau, Director of Cloud Data Architecture at Grainger (guest of episode #44) facilitated a discussion with Balvinder Khurana, Technical Principal and Global Data Community Lead at Thoughtworks (guest of episode #135), Carlos Saona, Chief Architect at eDreams ODIGEO (guest of episode #150), and Yushin Son, Chief Architect of Data Platform & Data Products Engineering at JPMorgan Chase. As per usual, all guests were only reflecting their own views.

The topic for this panel was an architect's view of data mesh, especially from an architecture lead standpoint. There are many challenges architects face in data mesh, managing the micro level minutiae, down to the data product output and input port decisions but balance that with crucial high-level decisions. Balancing the near-term and long-term vision and roadmap/North Star.

Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Scott's Top Takeaways:

1. Balvinder said, "…data mesh expects a lot more out of architects, there would be a lot of people and process and operation management that you will have to understand." It's important to understand that architects aren't just building out the systems but the entire org capability to do decentralized data well. It's a lot of responsibility but also a lot of interesting new challenges.
2. As keeps coming up in many episodes, doing decentralized data/data mesh doesn't mean everything is decentralized. That's what federated means in data mesh: building core building blocks that make the work far easier but not trying to chase into every corner on every use case - that's not scalable/repeatable. Creating ways for people to collaborate and interoperate easily. Doing the glue work so people can focus on the specific value add of the use case/data product.
3. It's absolutely crucial to understand that data mesh is not a complete vision just yet. If you are expecting to pick it up and simply run with it like it's a playbook, you're in for a bad time. The tooling isn't exactly there yet to do this easily and even if it were, we are still in the early days of learning the patterns to do it well. It's like microservices in 2013 more than 2023.
4. Every team will interpret data mesh differently - and many will interpret it in a way that lets them do what they want most :) Be prepared to step in to prevent people building everything themselves and also be prepared for teams that still expect a central data team to own everything. Communication and balance will be key.
5. Multiple years into your journey 'we're figuring it out' will still be a common refrain. Don't make flippant decisions but you'll learn and improve your processes so leave yourself maneuvering room but don't worry about getting things perfect. Optimize for learning and iterating to better, especially on the culture side.
6. Be prepared to compromise. You probably won't be able to decentralize major core systems - e.g. SAP ERP - immediately or maybe ever. While a perfect setup might be what you want, driving to value sooner is where to focus. Be practical and prepared for 'better but not good yet' kind of outcomes as you implement, especially early.
7. It's easy, especially early in your journey, to get overly focused on one domain or use case. That can be at the architecture level, the ways of working level, all sorts of governance aspects, etc. You MUST keep a balance between the micro - e.g. down to the use case or data product level - and the macro of can this decision scale and serve the broader organization.
8. Your organization is not a greenfield. Be prepared to not look exactly like any other orgs. You'll have lots of constraints. Frustrating yes, but every single implementation is messy behind the scenes. Everyone is trying to figure it out and cutting some corners and/or making missteps. Give yourself a break as you learn and iterate towards value.

Other Important Takeaways (many touch on similar points from different aspects):

1. We just plain don't really have data mesh best practices yet. They are starting emerge in certain aspects but if you aren't ready for some ambiguity and experimentation, you aren't ready to do data mesh.
2. Every organization will find different balances between centralization and decentralization for certain aspects of a data mesh journey. It's okay if your balance looks quite a bit different from others'.
3. Your balance between centralization and decentralization for many aspects of your implementation will shift over time. Constantly be measuring and assessing if things are 'good enough' and be prepared to change. But that also gives you the freedom to experiment :)
4. Pre data mesh, it was hard to conceive of doing decentralized data similar to how microservices works on the operational plane. It seemed it would cause chaos and data silos. But if done well, you can give teams significant autonomy and still have a larger internal data ecosystem where data from across the organization plays nicely and can be more easily used.
5. A centralized platform offering means more complexities than just one team owning a massive data lake setup - e.g. managing backwards and forwards compatibility across many domains - but also has lots of advantages. A biggie can be that with teams owning their own infra, there is no resource contention - especially important in financial services around month-end processing.
6. The old paradigm of central data ownership and control prevent fast releases so you don't really have any autonomous work by the producing domains. And you can't keep up with things changing - they all go through a backlog that means the time between something happening in the real world and the change to the table in the data lake can be weeks to months. That just doesn't work for the speed of change in the real world.
7. It's crucial to understand that data mesh is about building and using new muscles, ones around decentralization. Much like training, you will probably strain or pull some muscles. That's normal and to be expected. No one gets it perfect the first time around.
8. Building out the interoperability at the tooling level - not even the standards to interoperate the data but the tooling to make it possible for teams to combine data across data products or source data from upstream data products - is going to be harder than you expect. Not even policy/security governance, just the way data flows independently throughout the mesh rather than one giant pipeline per use case.
9. Early in your journey, you probably need to find domains that are excited to give data mesh a try. Rather than dragging a domain along, look for a true partner.
10. Some degree of failure is inevitable in a data mesh journey. But the cost of failure - and our ability to learn and iterate from 'failure' to 'not failure'/value - is so drastically different in data mesh. Fast failure, while possibly scary, is a crucial concept to embrace. Quickly learning and iterating is key.
11. Once you've built up the buy-in and excitement for data mesh, your job might become hype management :D People are used to data projects being built as fast as possible but with not as much focus on sustainability - data-as-a-product - or interoperability. Make sure to keep initial expectations manageable and share why doing it right is better than doing it right now.
12. When building out initial expectations, maintaining flexibility for breaking changes is very helpful. You don't want to make breaking changes carelessly but as you learn how to do data mesh, it's key to be able to scrap something that you find won't work rather than support a non-scalable platform for years to come.
13. Right now, you'll probably have to build a lot more tooling than you'd want, including some custom glue between services. Hopefully that gets far better but right now, there is still a lot of building to do if you want to do data mesh well.
14. If you weren't part of the microservices revolution and evolution, really study what worked and didn't. There are many lessons learned that you don't have to relearn :)
15. Yushin said, "Architects are not wizards. So we make a lot of choices and some of those choices will be wrong. And we need to experiment. And we need to pivot quickly once we find out that something's not going to work." That flexibility is necessary. If you don't have any flexibility to experiment and change, your org is probably not ready for data mesh.
16. It's still very difficult to abstract away the necessary capabilities for managing data from the tooling. It's very easy to get overly tied to specific tools. That is costly and difficult to find specialists for many tools and becomes a blocker to scaling. Look to build good abstractions where possible. Tools are hopefully coming soon.
17. Your room for experimentation is much broader, especially at the architecture/platform level, early in your journey. You probably don't only want to go with more 'legacy' tools, it's okay to try some bets on more 'cutting edge' offerings. There's balance of course but you can be a bit adventurous if you manage risk from that experimentation well.
18. Really focus on enabling self-service for the domain teams - there are certain aspects of governance that should be enforced automatically by the platform so teams don't have to reinvent everything and then it all becomes a mess because everyone is handling things differently. And it's okay to delay tackling use cases because you aren't ready to meet all their needs just yet.
19. It's going to take far longer than you'd like to sunset existing platforms and the data assets/data products using them. Don't press for lift and shift. Potentially look to the Strangler Fig Pattern as you replace existing use cases. And REALLY assure people you aren't going to decommission existing data platforms overly quickly.
20. Look to prevent copies of data that don't have a very clear reason to exist. If you can always get the data, do you need to persist it? There's storage costs yes, but a bigger cost might be the headaches around governance - including bad actor access risk - of keeping track of all those copies… As Carlos recommended, look to push the cost of that governance onto teams that want to keep copies and see if they still want to make copies of data :)
21. Potentially look to leverage tools that use open formats so you don't have to do custom integration around tools. Scott personal note: this is where I think data mesh can actually push the biggest industry changes. Imagine no more crazy proprietary formats and tools just work together!
22. You can drive data mesh participation buy-in from consumers by showing the general friction and pain points of your existing data setup. Almost every team has a lot of pain points. Can you really trust upstream data right now or do you want a contract that provides you a lot of assurances?
23. Cost will be a major factor in a lot of your architectural choices. Would that things were relatively equivalent but pricing across vendors is often extremely different. Be prepared to use more tools than you might want.
24. Different teams will have different capability levels regarding data. There is often going to be some needs that fall back to a centralized team, whether that is the platform team or another shared central team. In an ideal world, it would all be decentralized ownership but sometimes the cost of creating a full data function inside a domain isn't worth it. That's okay because that team shouldn't be a bottleneck if it's only handling a few teams but be careful 😅
25. Right now, a real value differentiation of data mesh is the ability to scale with nimbleness and agility while managing data quality. But for most organizations, at least right now, their data mesh implementation isn't super cost optimized or performant. Cost optimizations are typically second order concerns as you start out but will become bigger and bigger topics.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf