Is There a One Data Lake to Rule Them All?

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here. You can download their Data Mesh for Dummies e-book (info gated) here.

Kiran's LinkedIn: https://www.linkedin.com/in/kiran-prakash/

Kiran's article on the 'Curse of the Data Lake Monster': https://www.thoughtworks.com/insights/blog/curse-data-lake-monster

In this episode, Scott interviewed Kiran Prakash, Principal Engineer at Thoughtworks.

Some key takeaways/thoughts from Kiran's point of view:

1. ?Controversial?: You MUST have exec sponsorship to move forward with your data mesh implementation. You need the top-down push for necessary reorganization when the time comes. Scott note: only kinda controversial, really more often ignored :D
2. ?Controversial?: Data mesh, if done well, doesn't need to have a huge barrier to entry. That's a misconception. If you think about gradual improvement/evolution, you'll be on the right track.
3. "The Curse of the Data Lake Monster" was like the data field of dreams - there was expectation that if you build a great data lake, value will just happen. If you ingest and process as much as you can, the use cases will just happen. And it really wasn't the case. So we should apply product thinking to data to focus on what matters.
4. The 'Curse' was a manifestation of Conway's Law - the strong separation between IT and the business led to mismatched goals and subpar outcomes. With microservices, that started to be much less of an issue on the operational plane so why not try with data?
5. It's easy to lose sight of Conway's Law and aim for distributed architecture first but the organizations doing data mesh well are changing their architectural and cultural approaches and patterns together. Don't try to do the architecture first, you really don't know your key challenges yet.
6. It's very important to have a target operating model and get clear on your organizational vision and purpose around data - how will you use data?
7. Once you have an organizational vision and purpose, domains should start setting goals aligned to that vision and purpose.
8. As others have noted, don't get ahead of yourself, work in thin slices for your data mesh implementation. Stay balanced at an overall level between the data mesh principles as you add more and more thin slices but don't try to solve all problems up front.
9. If you modernize your legacy software but don't change the organization, expect to do the same type of modernization in about 5 years.
10. To really get to a scalable approach to data mesh, you should look for organizational and process reuse as much as tech/architectural and data reuse.
11. Move from measuring outputs to measuring value outcomes. Sounds simple - it isn't - and it's crucial to changing your mindset about how you approach data.
12. ?Controversial?: Use the 4 key metrics from DORA (https://dora.dev/) to measure how well you are doing in your software engineering. And a key aspect of data mesh is about applying good software engineering practices to data after all.
13. If you want to measure the value of your data work, you need to break it down into tangible objectives. Ask the owners of those objectives to provide the value of meeting the objectives. Then look to measure how much data work contributed to achieving those objectives.
14. Think of a use case as a value hypothesis: you are making a bet that something will have value. It's okay to be wrong, that's the nature of betting. But limit the scope of your mistakes so you learn and adjust towards value instead of making big mistakes.
15. ?Controversial?: If you don't have a culture where it's okay to fail, it will be very hard to do data mesh well. Scott note: it will essentially be impossible in my opinion
16. Many times what people consider minimum viable product is neither minimum nor viable. This is often due to a culture where you can't test things with users when they are still very rough. That will limit your success with data mesh. However, most people are reasonable so if you read them in that this will be a rough sketch/first iteration, they usually are on board to help you iterate to good.
17. "Architecture is about the important stuff. Whatever that is." - Ralph Johnson (via Martin Fowler)
18. Always think about necessary capabilities and build to those. The most important are those capabilities you need now. Don't get ahead of yourself.
19. The 'data platform' is really a misnomer, there will be multiple platforms. Users care about services, not if you have one platform or five or more. Don't have platform sprawl but don't over-centralize, that usually leads to scaling and flexibility challenges.

Kiran started off by talking about a blog post of his with a colleague from 2019 called "The Curse of the Data Lake Monster" - lots of clients were building big data lakes and it wasn't providing the expected value. There was an expectation that if you ingested and processed as much of your data as you could, it would create great use cases and lots of value. But it didn't happen. Value doesn't just happen without concerted and concentrated effort. So Kiran asked why aren't we applying product thinking to data to figure out and focus on what matters to drive value. What would happen if we focused on outcomes instead of platforms? What if we measured value not how many terabytes were processed and stored?

A key reason Kiran feels the 'Curse' happened was the strong separation between business and IT. That separation meant both were seeking different goals instead of collaborating. IT was focused on building things instead of solving business problems and the business side was focused on doing what they could, not building out a scalable and robust data practice. Conway's Law in action. We saw microservices really help to tackle those same issues on the operational plane so the data side of the house was definitely ripe for some product thinking-led innovation.

Data mesh avoids a lot of the issues of past data approaches by not leading with the technology - the first two principles are not tech focused. For Kiran, many (most?) of the organizations that are getting data mesh right are respecting Conway's Law and shifting their architecture and organizational approaches together. But in thin slices so as to not put too many eggs in one basket and make reasonable progress. And they are getting the exec sponsorship because while you don't want to reorganize your entire company upfront to do data mesh, you do need some top-down pushing to actually drive the necessary org changes when appropriate.

According to Kiran, while many people think data mesh has a high barrier to entry, that shouldn't be the case. There should definitely be a target operating model at the organizational level and organizations need to keep that in mind but it's not as though, again, you reorganize the organization all upfront. Organizations also need to really answer the question of what are they trying to do with data and what would doing data mesh well drive for them - if that's not crisp, they probably aren't ready to do data mesh because their business strategy isn't aligned to or contingent on doing data well.

Once the organization has the target operating model and vision down, Kiran recommends that domains should start to set their own specific goals aligned to the broader organizational vision. They should work on some hypotheses on how to achieve those goals and how they plan to measure their progress towards those goals. Start to build out your thin slice approach to making progress towards your vision and goals. Don't get super far ahead of yourself, look to progress at a meaningful but reasonable pace and tackle as little as is necessary now while still making sure you are aligning with the organizational vision. Keep your eyes on the prize, don't take on too much now. And yes, easier said than done.

Kiran pointed to a quote he read about if you modernize your legacy software stack but don't change your organization, you will need to do the same modernization in about five years. The same goes for data - if you are taking on data mesh from a tech-first approach, you'll just have to do all the same modernization again and you won't get a lot of the potential benefit from data mesh. Decentralizing the architecture will only really change things if you change the organizational aspects too. People, process, technology.

For Kiran, we need to really start to focus more on measuring value outcomes instead of inputs - how many terabytes or operations per second isn't directly tied to value. Teams need to have it made clear what is actually valued and valuable. In many large organizations, there are often less clear links between data work and business value so you have to educate and incentivize teams to do the high-value data work.

When thinking about trying to measure the return on investment in data work, especially data mesh, Kiran recommends starting by breaking it down into more tangible goals and measuring the value of achieving those goals. Then you can start to say how did the data work contribute to achieving those goals. But a data team can't really know the value themselves, whether inside the domain or not. And by breaking things into smaller goals and objectives, you can more quickly iterate towards value with tight feedback loops. Instead of large-scale projects, you build to larger and larger objectives through breaking things down and achieving meaningful micro progress that leads to large macro value.

Kiran talked about thinking of use cases as value hypotheses: you believe it will have value and thus you are making a bet. And it's okay to get things wrong, just limit the scope of the mistake so you can use the missteps as learning so the larger macro bet has a much higher chance of paying off. This is iterating to value, those tight feedback loops. If you don't have a culture where it's okay to be wrong, okay to fail, then data mesh is potentially (Scott note: almost definitely) not right for you.

Minimum viable product is often neither minimum nor viable in Kiran's experience. If you can't put something pretty rough in front of stakeholders, you waste far more time and effort building in wrong directions and are less likely to hit on success. But that's often out of the control of the product team. So it's a catch-22: do you put in a lot of effort to get it well past MVP or do you risk losing face? So we need a culture where we can actually do thin slicing well to really derive the most value out of building software, whether that is apps or data products. That incremental value delivery is really crucial to maintaining nimbleness as you scale. If you can't actually deliver in thin slices, it can significantly increase risk as you are making larger bets. But Kiran's seen that if you spend the time to explain the need for thin slicing and that what they are looking at is the 'sneak peak' and you just want feedback, most people get it and are reasonable. But you need to communicate about it :)

Kiran used a phrase Martin Fowler uses often from Ralph Johnson: "Architecture is about the important stuff. Whatever that is." When thinking about decisions that are hard to reverse, spend a lot more time and care, but the ones where it's easy to reverse, those probably don't make up the core of your architecture. When building your architecture, it's again important to build incrementally instead of trying to get it perfect from the start. Think about necessary capabilities, not technologies. In data mesh, that is about data product production and then monitoring/observing, data product consumption, mesh level interoperability/querying, etc.* Start to map out what you need and then think what level you need from a capability standpoint and when. You don't need to build out capabilities for when you have 20 data products when you have 1-2 data products.

* Scott note: Kiran went on for quite a bit here about necessary capabilities for data mesh late in the episode if you want to listen.

Quick tidbit:

Leverage the 4 key metrics from DORA to measure how well you are doing your software engineering as applied to data - 1) Lead Time to Changes (LTTC); 2) Deployment Frequency (DF); 3) Mean Time To Recovery (MTTR); and 4) Change Failure Rate (CFR) https://dora.dev/

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf