How to Automate Data Quality Work

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts (most interviews from #32 on) here

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here. You can download their Data Mesh for Dummies e-book (info gated) here.

Lynn's LinkedIn: https://www.linkedin.com/in/lnoel/

DAMA New England: https://damanewengland.org/

Correlation/Causation XKCD: https://xkcd.com/552/

In this episode, Scott interviewed Lynn Noel (pronounced Knoll), Data Governance Lead at AIM Consulting. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Lynn's point of view:

1. There is a strong data governance value proposition in data mesh around making data broadly "safer, better, and easier to use and share." Lean into that value and communicate it widely.
2. ?Controversial?: Many organizations doing data mesh are moving from a very centralized approach and they have certain concerns and approaches they should take. While other organizations are overly decentralized and need to be using different approaches to migrating. Really consider these as two different approaches that land on a similar end approach.
3. You should look to return governance to its original meaning: guiding and steering. In data mesh, governance shouldn't be about control.
4. ?Controversial?: "Not all governance is created equal." There are non-negotiables: infrastructure security and data classification. Don't over-index on sharing data at the expense of crucial governance.
5. Shift your governance left. Shift left the responsibilities - where appropriate - but also shift governance left in your timelines - it needs to be part of the build processes.
6. If you are doing data mesh, you have to understand the existing organization governance structure - whether that is called data governance specifically or not - and align with it. Legal, risk management, IT, etc. Make sure they are aware of your work and give them confidence you will keep things governed well.
7. Every member of a project team on a consulting engagement is obligated to deliver value. That means you need to translate your work to business value. Governance isn't only risk mitigation, it should have incremental value.
8. In data, we too often "confuse the pipes for the water" - meaning we focus too much on the plumbing instead of the resource people want: understandable, usable, trustable data.
9. Consider telling business users you are providing them an ecosystem of capabilities instead of a platform. They can understand that they have their own "habitat" and can customize aspects to meet their own needs but that it all needs to work well together too with other habitats.
10. Federated computational governance is an overly scary term. Break that down for users and what it means for them and why it actually makes their lives easier/better.
11. ?Controversial?: "If you automate something that users aren't expecting to have automated, they will be extremely surprised and disoriented by the platform." Make sure to communicate and ask people what they actually want automated. What you see as friction might be value-add work for them. Communicate!
12. You can plan out a lot of things but they just won't survive "first contact" with the data model and/or the business model. Be prepared to iterate, don't plan too much and be flexible; iteration is great, you are implementing what new you've learned.
13. If your organization is coming to data mesh from a highly decentralized approach, it's very likely there are pockets of data that never were put into the central data infrastructure. It will probably be hard to get people to trust sharing that data and giving up control of it because the previous approach couldn't handle what they were doing.
14. ?Controversial?: Part of data mesh or any transformation initiative, you need people who have ownership over, the responsibility for, organizational change management. Driving buy-in at the personal and organizational level. Scott note: I kind of like this but it's also scary. Who has the responsibility of driving buy-in specifically?
15. Use the phrasing capability instead of feature when talking data platform. What capabilities are you offering to enable your use cases? And your capabilities should be very use-case agnostic.

Lynn started with her being the lead data governance person for a data mesh implementation at a client. It's allowed her to have governance be part of the value drivers for all aspects of mesh. She said data governance in mesh is about making data "safer, better, and easier to use and share." The value proposition of doing governance well in data mesh starts to emerge when you think that to really get the most out of data, you need teams to feel like they can rely on data, not just use it.

It's crucial in all aspects of data, but especially with data mesh, that governance is an enabler instead of a blocker according to Lynn. As a governance leader, you can't be trying to put up blockers but it's crucial for the team to understand why aspects of governance are helpful and add value. You should be building alongside - especially the building aspect, being part of the work - to keep things moving while also making sure you aren't setting yourself up for trouble later, whether that is quality, regulatory, or otherwise.

Lynn said "not all governance is created equal" - it's important to establish where you can and cannot compromise. Infrastructure security and data classification are both non-negotiables. Both wrap in to external threats and internal access control, including to regulatory requirements. Risk management and compliance just cannot be ignored so make them part of the build process and ensure everyone understands why they matter. It's a core of product thinking around data.

There are multiple ways to 'shift left' in governance in Lynn's view. The first is shifting governance left in the project timeline - it should be part of every aspect of building your data mesh from the start - platform, data products, etc. You don't layer governance on at the end. We also should be shifting some of the responsibilities left to the developers - while giving them the capabilities and knowledge to handle what they can and come to experts for what they can't.

Lynn has had some success with talking to business users about not providing them a platform but an "ecosystem". It lets them envision something on a bit more of a tangible level, that they will build out their own habitat but it plays into a larger ecosystem. So they are comfortable in their own habitat which lets them better deliver value to themselves and others. Of course, your ecosystem can't infringe on other ecosystems so it is a good analogy for how domains all interact.

When looking at what to automate as part of your platform, Lynn recommends strong communication with users. You can surprise and disorient them if you automate the wrong things. E.g. there are certain aspects of discovery analysis that look a lot like standard tests but one is incremental work where you want to dive deeper into the data and one is part of daily work without value add from doing manually. Automate the second but not the first :)

Lynn brought up that your organizational setup before moving to data mesh will have a very large impact on what you should focus on when in your journey. If you are coming from an overly centralized approach, it will mean that you understand the benefits of integration but might have some challenges with letting go of central control. If you are coming from a very highly decentralized world, you might need to focus much more on alignment because domains may have had freedom to do as they please with little thought for users in other domains.

In organizations doing data mesh coming from a very highly centralized approach, Lynn says be prepared for a lot of dark data. Because the officially blessed IT systems were overly rigid, many people were doing work outside of the systems, often in Excel or similar. Getting people to trust you that you'll take good care of their data and they can still maintain access will be hard. There are many fun governance challenges including naming conventions to tackle with data like this.

For Lynn and her team, it's been very important to focus specifically on answering what's in it for me from users. Me in the personal sense and me in the organizational and/or domain sense too. And having people whose role involves owning organizational change management is very helpful as part of a product team. There is literally a person or people responsible for getting alignment and driving buy-in.

Quick Tidbits:

In data, we too often "confuse the pipes for the water" - losing the forest for the trees. We focus too much on the plumbing instead of the resource people want: understandable, usable, trustable data. Focus on the outcomes and business value.

It's easy to scare people by using the 'federated computational governance' term. Work with teams to show them where we can automate the past pains of governance so everyone can work on what drives incremental value not manually applying policies.

Regarding a data warehouse: "I never saw a plain vanilla implementation survive anybody's business data model." Your plan will not survive first contact with the business model as is. Be prepared to iterate on your plans in data mesh. But now, that doesn't break everything like it did for the enterprise data warehouse.

Similarly "no taxonomy survives first contact with the data." You have to build so you can update and iterate, prevent brittleness.

Don't use the phrase feature for your platform, use 'capability' instead. Make sure you are building the capabilities to support multiple use-cases and start to prioritize capabilities based on what use cases they can unlock.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him at community at datameshlearning.com or on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf