Data Mesh Radio

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/Please Rate and Review us on your podcast app of choice!If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see hereEpisode list and links to all available episode transcripts here.Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center hereGoDataDriven Data Mesh Webinar: https://godatadriven.com/topic/webinar-data-mesh-9-feb-2022-thanks/GoDataDriven Self-Service Whitepaper (info-gated): https://godatadriven.com/topic/data-democratization-whitepaper/Steven's LinkedIn: https://www.linkedin.com/in/stevennooijen/Guillermo's LinkedIn: https://www.linkedin.com/in/guillermo-s%C3%A1nchez-dionis/In this episode, Scott interviewed two people from the European data consultancy GoDataDriven - Steven Nooijen, Head of Strategy, and Guillermo Sánchez, Analytics Engineering Tech Lead. Guillermo started off by talking about how for the last ~3 years, he was seeing the data engineering team as the bottleneck before data mesh came onto the scene. For Steven, they were seeing lots of companies that were building out data platforms, especially data lakes, and then not really getting the promised benefits so data mesh made sense. All agreed data mesh is not right for every company and then mentioned some good signs that an organization should consider data mesh. Guillermo pointed to a lot of the usual suspects: size of company, size of data team, how many data consuming teams do you have, how many data sources do you have, etc. He then gave a specific example: if you have a data analyst in a consuming domain that has to wait more than 1 week for data, there is a bottleneck somewhere. Is it centralization? Not sure but time to investigate and that might be where you start to consider data mesh. Steven gave the example of an even earlier indicator that bottlenecks are occurring: teams start to hire their own data people rather than leverage the central team. Guillermo also pointed to the rise of consuming teams getting direct data access from producing teams instead of going through the data team.Guillermo made a very crucial point: data mesh is really about interfaces. People talk about data products being the technical communication interfaces between teams. So as long as people trust the data product and teams adhere to interface standards, data mesh can work. But that trust is earned, not given. Both Guillermo and Steven talked about the need for an easy path, a golden path. With that golden path, it can be relatively simple and low friction for domain teams to share their data for the vast majority of use cases. There is also a need for some quality control to make sure data products are trustworthy, have reasonable SLAs, etc.Per Steven, what they've seen to date at customers is most domains are happy to follow that golden path. But, you certainly need that path to start out where they are - trying to make them use foreign tooling/workflows is going to be a tough sell. They are also leveraging the concept of a "certified" data product label. The core data mesh implementation team awards this to high-quality data products and it's easy for teams using the happy path get - it is harder to get certified if you use your own tooling. This acts as a quality control measure. Steven talked about when evaluating where on the centralization/decentralization slider decisions should fall, you should really look at domain maturity and organization maturity. Centralization provides more support, if being more of a bottleneck, in many cases so there is a trade-off. Are domains really ready to take on the full responsibility of whatever task you are considering? It's okay to not have a one-size-fits-all approach.Somewhat atypically, Guillermo and Steven recommended starting your proof of concept, if you need to do one, with a single domain. They recommended starting with an eager domain and to make sure they have support from the platform team. Pick a consumer-aligned domain with a high demand for data. Again, this advice runs somewhat counter to what many have said and that is absolutely valid.Instead of the proof of concept, Steven recommends driving high level buy-in via a C-level sponsor. That way, you have time and funding to get things moving. Obviously easier said than done but if you can go this route, your journey is likely to be smoother as you won't have to constantly be looking for funding. When driving buy-in via the C-level sponsor, Steven mentioned never once using the phrase "data mesh". The execs really responded to using "self-service" and the 4th industrial revolution concept.When driving buy-in, Guillermo talked about directly telling the teams what does this mean explicitly for them. You have to make it mutually beneficial! Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see hereAll music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/A mesh musings asking if we should embed data/analytics engineers into the domains to serve as the data product developers or not.Please Rate and Review us on your podcast app of choice!If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see hereEpisode list and links to all available episode transcripts here.Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/Please Rate and Review us on your podcast app of choice!If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see hereEpisode list and links to all available episode transcripts here.Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.Data Mesh Learning Meetup Presentation: https://www.youtube.com/watch?v=7iazNKG8XQoSarita's LinkedIn: https://www.linkedin.com/in/saritabakst/In this episode, Scott interviewed Sarita Bakst, Managing Director and a leader in Firm-wide Data Management at JP Morgan Chase. Sarita was previously on a Data Mesh Learning meetup and is helping lead the firm's data mesh governance charge.Per Sarita, historically, data governance has meant controls and gatekeeping to most people - typically getting in way of innovation. So there needs to be a focus on changing that narrative, not just through words but actions to show it's not the case. In data mesh, you need to ensure that domains can make good decisions on governance and seek out subject matter experts when it makes sense. Sarita covered that one of the key issues in the way governance has been done is the people making decisions - the central governance team - don't have the real understanding of the data. When those decisions are put in the hands of the people who really know the data, but with guardrails and guidance, the fear is lifted about can we actually use this data and how. This opens up lots of new opportunities to leverage your data.Sarita strongly recommends starting with purpose-built data products. Find a use case and build data products to serve that specific. And data products MUST be about unlocking business value. You don't need to serve up all of a domain's data on day one, in version one of that domain's first data product - make it extendible and reusable so you can find additional consumers and expand over time.Get out of your own way on data governance in data mesh. You are going to learn and your approach will evolve as you learn. It's okay to not know everything upfront, set yourself up to not get in trouble - put the proper guardrails in place - but you won't know everything. Think of designing your risk controls as toll-gates and make sure they aren't bottlenecks.Have standards (not standardization) so people don't have to invent things from scratch. Standards for interoperability, naming, etc. - think of them as guiding principles instead of rules. Make sure domain owners know who to contact and when on governance subject matter expertise. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see hereAll music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/Please Rate and Review us on your podcast app of choice!If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see hereEpisode list and links to all available episode transcripts here.Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.What the Heck is a Data Mesh?! Post: https://cnr.sh/essays/what-the-heck-data-meshChris' Twitter: https://twitter.com/criccominiChris' LinkedIn: https://www.linkedin.com/in/riccomini/Chris' website: https://cnr.sh/The Missing README book: https://themissingreadme.com/In this episode, Scott interviewed Chris Riccomini, a Software Engineer, Author, and Investor. Chris led the infrastructure team at WePay when they embarked on a data mesh journey and made a well-written post on thinking about data mesh in DevOps terms.Like a number of people/organizations that have come on the podcast, at WePay, Chris was pursuing the general goals of data mesh and was applying some of the approaches as well - but it was not nearly as cohesive as Zhamak laid things out.Their initial setup had two teams managing the pipeline/transformation infrastructure. Chris's team was mostly handing the extracting and loading and then there was a team of analytics engineers handling the transformations. The Transformation team saw a major increase in demand and quickly became overloaded -> a bottleneck. Chris' team also started to get overloaded so they knew they had to evolve.One way the team started to address the bottlenecks was by decentralizing the pipelines. Teams could make a request and a scalable and reliable pipeline would essentially get automatically set up for them. WePay is in the financial services space so as part of those pipelines, to prevent risk, teams could mark their sensitive/PII columns and the infra team also put in some autodetection capabilities to make sure they didn't miss any.WePay created a "canonical data representation" or CDR, which is pretty analogous to a data product in data mesh. Chris really liked WePay's use of the embedded analytics engineer to serve as a data product developer.One key innovation for WePay was tooling to enable safe application schema evolution. They looked for things like dropped columns and had more comprehensive data contract checking mechanisms. It allowed developers to test changes pre-commit. 80-90% of data breakages were things the developers had no idea would cause an issue and they reverted those changes. 10-20% of the time, the developers still wanted to go through with the changes and that kicked off negotiations with data consumers. That forced conversation was very helpful for a few reasons. Chris talked about standardizing around technologies for the platform but allowing teams to roll their own if they wanted. But they were super clear with those teams that the infra team wouldn't support those other technologies, even if it was okay to use it. He also sees a major need for an API gateway concept for data. Currently, everything around versioning, auto-documentation, etc. is way too manual and high friction. Chris talked about taking the learning from DevOps and applying them to data mesh. One good one to look at, per Chris, is the embedded SRE concept - should you do the same with a data or analytics engineer? There is also a need for many standards and replicable patterns. They launched a data review committee, similar to a design review committee, that helped come up with standard data models and other standards.Sherpas not gatekeepers - build out your review functions as councils to guide and disseminate knowledge. The team's role should be about assisting where they can, being a trusted partner. And what WePay saw was as people went through more reviews and similar, they saw there was less of a need for them as people learned what good/best practices were.Lastly, Chris wrapped on a point others have made about the cost of making a mistake. You need to make it as low as possible. Back processing data can be very costly in time and compute.Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see hereAll music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/Please Rate and Review us on your podcast app of choice!If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see hereEpisode list and links to all available episode transcripts here.Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/Please Rate and Review us on your podcast app of choice!If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see hereEpisode list and links to all available episode transcripts here.Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center hereMarius' LinkedIn: https://www.linkedin.com/in/marius-ingjer-a155313/Marius' Medium: https://medium.com/@marius.ingjerIn this episode, Scott interviewed Marius Ingjer, Co-founder and Senior Consultant at Knirkefritt AS. Marius has been working with a few clients on implementing data mesh. They covered 3 distinct topics: evaluating if data mesh is a fit for your organization, team structure challenges in data mesh or data mesh-like implementations, and a simplified definition of federated computational governance.Evaluating if data mesh is right for you:To start, Marius provided a list of evaluation questions to help you determine if data mesh might be right for you: How many data sets are you producing?What is the lead time to creating a new dataset?How well are your datasets serving your data needs?How many domains do you have?How complex are your domains?How does the team respond to new data requirements?How usable in general is your data?Every company wants to share data well but the centralized data team isn't the bottleneck yet for many. Centralization can add a lot of value until it starts to become more hurtful than helpful and yes, figuring out that point is easier said than done. Centralization of data fights Conway's Law and can become way too much cognitive load so it will eventually become an issue for many organizations.A key question in evaluating if data mesh is a fit: what is the cost of allowing your data processes to fail? Per Marius, the business consequence of failed reports has historically not been that high. But if you are driving business decisions, whether that is ML or just crucial day-to-day decisions on your data, data mesh might become more attractive.Team structures and challenges in data mesh:In general, it's important to understand that implementing data mesh will cause cultural challenges - Marius believes developers generally don't want to ALSO share their data. It's additional work so you have to align incentives, which is far easier said than done. That additional cognitive load on developer plates is very crucial. We need to make we address that load to not burn them out. That means realigning incentives but also having extra help with things like grooming the work backlog. Providing extra resources helps but that is more about tackling the work, not handling the increased cognitive load. And learning about how to do data well is a pretty big learning task.Marius recommends giving teams the extra resources but also reshaping the team and business structure, such as the KPIs, to effectively prioritize and shape the requirements. He also recommends having a stick, not just carrots, or teams will try to just opt out.Simplified definition of federated computational governance:When talking about this topic with developers right now, it feels far too complex. To make it less complex, reduce the friction to developer decisions but not add much to cognitive load. An example might be providing easy data masking tooling for PII or extensible data APIs so developers can focus on the value-add.Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see hereAll music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/An episode about how to figure out what is good to reuse from your existing data approach and what to toss out when it comes to data mesh.Please Rate and Review us on your podcast app of choice!If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see hereEpisode list and links to all available episode transcripts here.Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/Please Rate and Review us on your podcast app of choice!If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see hereEpisode list and links to all available episode transcripts here.Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center hereScott Hawkins' LinkedIn: https://www.linkedin.com/in/scott-hawkins-8934393/In this episode, Scott interviewed Scott Hawkins, Principal Data Architect at ITV. Scott views data mesh as a mechanism for change. Your company culture and your understanding of it are crucial to establishing data mesh well, driving that buy-in. Ask yourself: what challenges does data mesh actually address and hopefully solve, how will it impact the business not the tech, and what does it change. Your organization might not be ready for data mesh. Or a specific domain might not be ready. And that's okay! As ITV moved forward, they found a "good enough" solution via a global ID. It's not perfect, there might be some overlap - such as one person might have a different global ID for their online subscription versus their broadcast subscription - but it is far better than what they were doing. And it allows for interoperability/joins across the data. This is a big improvement - don't let perfect be the enemy of good or done.One thing working for ITV is deploying a "team-in-a-box" to help domains move forward - similar to an internal consulting team. Each situation is different so each box they are given is different. The team-in-a-box concept also means it is somewhat easier to build common best practices internally. Coming to the table with defaults has really helped ITV. Per Scott, there are 3 good ways to drive buy-in for the domain teams: At the senior level - so it trickles down as the management for the team is bought in.Via a strong carrot - solve a problem for them as a kind of quid quo pro / mutually beneficial solution. Trying to solve an unrelated problem will drive lower buy-in.Work on realigning the team KPIs/OKRs with the senior leaders to actually realign incentives. Continuing on the driving buy-in, Scott recommends working with the domain managers to generate a viable/valuable carrot for the entire team. Explain to those leaders why it matters, work with the leaders to revamp the KPIs if the KPIs are getting in the way of delivering a good data product. This is why exec-level buy-in is so crucial - it is pretty hard to start modifying team KPIs/OKRs without it! Talking to teams to understand who they (the individual and the group) operate is crucial to developing the right path for them.Scott also talked about making failure an option. You can try to work with a domain and if it isn't working, it's okay to move on. You don't need to get everyone onboard on day one or sharing their data on day one. If you design incentives well, people will want to participate eventually. Until then, it's okay to walk away from that team.Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see hereAll music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Lorenzo Nicora, Principal Data Consultant at Mesh-AI, discusses skipping the fluff of Domain Driven Design for Data Mesh. They explore starting with manageable problems, challenges of modeling data in different domains, importance of language in data modeling, managing time zones and interoperability, understanding domain-driven design, starting with manageable communication, and services offered by their firm.

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/Please Rate and Review us on your podcast app of choice!If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see hereEpisode list and links to all available episode transcripts here.Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf