How to Build a Data Mesh With Multiple Domains

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts (most interviews from #32 on) here

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here. You can download their Data Mesh for Dummies e-book (info gated) here.

Chris' LinkedIn: https://www.linkedin.com/in/christopher-andr%C3%A9-haas/

Article on Lean Value Tree: https://rolandbutler.medium.com/what-is-the-lean-value-tree-e90d06328f09

In this episode, Scott interviewed Chris Haas, Advisory Consultant at Thoughtworks. To be clear, he was only representing his own views on the episode.

Some key takeaways/thoughts from Chris' point of view:

1. When starting a data mesh journey, you have to find teams that are actually willing to participate. If they won't truly take ownership of their data, it's essentially a non-starter. You can help convince many domains by showing them the investment you'll put into their teams on upskilling and other capabilities enhancements - it's not just new responsibilities and work.
2. In data mesh, look to drive domain-wide understanding and buy-in, at least to the vision of what a more data capable domain means, the benefits to the domain. Not everyone will or has to care about data mesh specifically but you shouldn't have data mesh by decree from upper management.
3. Consider lean value tree mapping - it helps everyone align on a single mission with goals to support the mission, what bets you have to make for each goal, and on down. It can help you stay focused on what are you trying to achieve.
4. For use case prioritization within a domain, look to 1) what are the business goals, 2) what use case will have the highest value, and 3) what is the easiest to execute on. Don't get overly focused on value if you aren't ready to actually deliver it. Scott note: don't forget risk too
5. ?Controversial?: Choose a use case with a single domain if possible for your organization's first data mesh use case. It just limits the amount of challenges and potential failure modes.
6. It's common to want to first focus on the technical aspects of data mesh - it's the tangible part and easier to see changes - but you should be focusing on the organizational aspects just as much if not more early in your journey.
7. You can start your data mesh platform team in a central data team, that's a normal pattern. And it doesn't have to be huge, 4-5 people building and a product owner - the platform is a product too.
8. A domain-based data product team doesn’t have to be large or require a big reorganization of the domain. Make sure they have the capabilities - data fluency and domain subject matter expertise - to execute but it can be a small team carved out of the domain.
9. Data product teams tend to be long standing teams but they are typically responsible for developing multiple data products, not just a singular data product. Most data products don't need that level of day-to-day tending :)
10. ?Controversial?: You should consider your data product team a new team in the domain and a new team means budget. If you don't have budget and you don't rework priorities, you are putting additional work on a domain and the likelihood of success falls - look to avoid that risk.
11. Doing decentralized data, doing data mesh doesn't mean everything is decentralized. Scott note: this is a persistent myth around data mesh. Not everything becomes decentralized or you will just create silos.
12. ?Controversial?: Once you've proved out you can do data mesh and drive value, when thinking about adding new domains, consider setting up a transformation office to coordinate, upskill, and prioritize work across domains.
13. Look to build an internal community to encourage cross domain communication and hopefully collaboration.
14. Finding cross-domain use cases is likely to be more difficult. Leverage that internal community to create more conversations that result in new, valuable use cases.
15. When finding your data products for a specific use case, start from what would be the specific data product to support this use case. Then start to work backwards towards what source-aligned data products you need. And of course, look at existing data products for potential reuse first.
16. !Controversial!: Source aligned data products should be - or at least start off - "very specific and very small."

Chris started out with a rather blunt but crucial statement: when thinking about data mesh, you have to identify at least one domain that will actually take ownership of their data. A successful data mesh implementation can't be entirely IT driven. And domain data ownership and coupling that with data as a product are typically very much not the natural order for most large organizations. You will need to invest into domains to make them capable of owning their data - showing that you will invest in their success can help win them over - so you want to make sure it will be money/resources/time well spent.

For Chris, it's very important for there to be at least a domain-wide understanding - and hopefully buy-in too - for what the domain is trying to do around data. That can be about data mesh or simply how they are changing their relationship to data. It won't work well to do data mesh from just upper management buy-in and pushing that down as a mandate. And that requirement can lead to deciding that data mesh isn't right for a domain and that is perfectly okay and reasonable - not every data management challenge is a nail for a data mesh hammer.

Lean value trees are an important tool for Chris and team when speaking with a domain about data mesh. What are they actually trying to achieve - the mission - and work backwards from there. Break down the goals, then figure out the assumptions or bets around those goals. This helps you stay focused on what you are trying to achieve - is it deliver a data product or is it address the use case and create business value?

So the output of a lean value tree helps align the team on a mission or missions which align to the business goals. Then, when thinking about use case prioritization, you need to balance those business goals, the amount of expected value of a use case, and the expected amount of work to deliver that use case*.

* Scott note: it is kind of included in the amount of work and expected value but I'd also factor in risk - what is the risk of this use case not being valuable and what is the risk of the team not being able to execute well on the specific use case. Probably don't take a big gamble on your first use case ;)

At the start of a journey, Chris recommends to find a use case that benefits the producing domain if at all possible. Yes, we want domains to publish data to benefit the entire organization but if a domain is going to be the test subject and invest their money, their people's time, their people's cognitive load, etc., it will be hard to find a domain willing to do that for another domain without decently strong incentivization. And at the start of your journey, those incentivization and community mechanisms are likely hard to come by. If you don't have these challenges and domains are all happy to help each other, consider yourself very lucky.

Based on interactions with a number of clients and prospects, many organizations would rather focus on the technical aspects of data mesh first over the organizational and Chris wishes that weren't the case. While building out the technical aspects is no easy task, if you aren't ready to actually do the day-to-day work, what are you building the tech to support? Scott note: this is extremely common and is also a very common comment from consultants. The tech feels more tangible and it's easy to say yes/no than squishy operating model discussions. But they are crucial to doing data mesh right.

Chris believes an early data mesh alignment on the organizational model doesn't have to be - and shouldn't be - disruptive. You can have a domain start to carve out new ways of working without doing a major reorganization. There should be a team building the platform and a product owner for the platform but 4-5 people building is reasonable and they can live in a central data team, that's a normal pattern, no reorg needed. As for the data product team, you want them to be part of the domain if possible and it should be a mix of highly data fluent people and domain subject matter experts but again, it can be a small team. So a small team carve out but not realigning the entire domain. Scott note: remember domain is an overloaded term. Some domains are sub-domains that are 3-5 people but typically we mean the line of business which can be 10K+ people.

For Chris, when asked about data product teams, he said you should look to have them be long-standing teams. You need an owner per data product but often the development teams will be charged with creating multiple data products rather than only one. Especially as you build the knowledge of how to build good data products, that team can become more efficient. But do not treat your data products like projects, they should be managed/evolved by long-standing teams, otherwise they are more likely to fall to data disrepair.

As a consultant, Chris recommends working with a consultancy :D but the point is that your data product team in the domain should be considered a new team and that new team needs a budget. It might be reallocated budget but hopefully it's new budget. But the consultancy angle is that if the implementation doesn't deliver value, it's easier to cut the consultancy than reassign or lay-off employees. Scott note: I think this is a valid concern but many can't get the budget and many organizations are doing this work entirely internally.

Once you've proven value from and a capability to do data mesh, a typical pattern is to set up a transformation office that will assist other domains in their journey according to Chris. That way you can maintain coordination, find reusable patterns - tech, architecture, people, process, etc. patterns -, prioritize, upskill, etc. That office is helpful in getting teams up to speed but also setting expectations - domains won't magically transform their data capabilities overnight. And look to build an internal community function to encourage cross domain communication and hopefully collaboration.

For Chris, that community aspect is really crucial to identifying cross-domain use cases. The data product owners should be communicating with each other to discuss what they've created in case that sparks new ideas for new use cases. For finding what data products you need to support a use case, start from the mission, then the consumer-aligned data product that would best serve that mission. Once you know what you'd want to have as the end product, you can start to find the necessary source-aligned data products.

Chris wrapped up with a few useful tidbits:

Don't worry about trying to serve future, unknown use cases with data products you are building now. Build for reuse and build for evolution and then evolve when those new use cases emerge.

Source-aligned data products should be - or at least start out - "very specific and very small." Don't try to cram too much in. Scott note: I think this can create some discoverability issues but it is a pattern I am seeing more. See Carlos Saona episode #150 for how that looks in an implementation

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf