Decentralizing Data: From Data Monolith to Data Mesh with Zhamak Dehghani

Barry O’Reilly and this week’s guest, Zhamak Dehghani met 10 years ago when they worked together at ThoughtWorks. Zhamak is currently the Director of Emerging Technologies at ThoughtWorks and the creator of Data Mesh, which Barry describes as “one of the most exciting paradigm shifts in how we manage data at scale.” He and Zhamak discuss why traditional data architecture models are failing and how applying product thinking principles to data management is a way to harvest the data’s full potential. “This show,” Barry remarks, “is for those who are curious to understand how to bring the convergence of product thinking, data management, and distributed systems development together to create platforms and products of the future.” Early Values Zhamak has always believed in distribution of responsibility and decentralization of ownership. She finds that these design principles are more compatible with real life. Colleagues taught her the Unix philosophy early in her career which now forms the basis of her data management approach. “They taught me those wonderful ideas to build systems and programs that do one thing and one thing really well. But most importantly they work together really well,” Zhamak says. “‘Simple is beautiful and beauty is the truth’... Reduce systems to their simple principles; then together can emerge complex behaviors.” She saw an opportunity to bring the UNIX principles to data. Challenging Assumptions It often takes someone new to a system to point out obvious flaws to long-time practitioners. Zhamak says that when she came into the world of big data, she was agnostic to the accepted assumptions, so she felt free to challenge them and conceive a different paradigm. For some reason when it comes to data, people eschew UNIX principles and see it as something to be centralized. Unsurprisingly, a data lake becomes monolithic and departments become siloed. Reimagining the world of data requires a new language, she points out: “The moment you need to imagine something different you need to use a very different language.” Instead of seeing data as an asset - which you want to hoard and get more of - Zhamak advocates that data can be seen as a product which should be used to serve internal and external customers. Barry adds that the idea of the single source of golden data makes companies unable to move as they get bigger. Move to Product Thinking Barry comments that the shift towards product thinking started with Amazon. Their monolithic database was preventing them from scaling. “They realized that they needed to create these smaller, more autonomous units that had the capabilities to build things just like product teams. This is where this notion started to emerge from changing the organizational design... both technically and just how teams would work together,” Barry says. In this new way of working, teams could experiment and own outcomes. They could make small, quick changes and see the effects.   What is Data Mesh? Instead of trying to fit data into a mold, Zhamak feels that its dynamism should be embraced. “Create an architecture and ownership of data that starts with the assumption that data can be useful and shareable and trustworthy right at the point of origin; and then allow for different domains and different aggregations, different projections to get created as a mesh picture,” she posits. She explains how this new view of data impacts ways of working and the type of platform a company would create. The four principles of the Data Mesh philosophy are, “domain ownership of the data; data as a product; self serve data platform to enable autonomous teams; and a federated computational governance to balance the interoperability of a decentralized world with the trust and security built-in.” Read the rest of the show notes on BarryO'Reilly.com Resources Zhamak Dehghani on LinkedIn | Twitter Data Monolith to Data Mesh article Data Mesh Principles article