How to Have a Conversation About Metrics

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here. You can download their Data Mesh for Dummies e-book (info gated) here.

Frannie's LinkedIn: https://www.linkedin.com/in/frannie-farnaz-h-a7a11014/

Post on the Product Trio concept by Teresa Torres: https://www.producttalk.org/2021/05/product-trio/

In this episode, Scott interviewed Frannie Helforoush, Technical Product Manager/Data Product Manager at RBC Global Asset Management. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Frannie's point of view:

1. There is a difference between the product mindset and creating/maintaining data products but both are very important to exchanging value through data. We should be looking to apply the product mindset to all aspects of our data work, not just how it applies to data products specifically.
2. To do data product management well, you should look to software product management practices and recontextualize those to data. Many map well but some don't. It's not a copy paste, think through what should be applied to data differently.
3. The data product manager needs to serve as the bridge between data producers and consumers, making sure consumer requirements are satisfied much like with a software product manager where they are the bridge between software engineering and software users.
4. If product management is the intersection of business, tech, and user experience (UX), how should we think about that for data product management? Tech and business are easy but there isn't a user interface (UI) so think of UX in terms of data fluency, access, and documentation.
5. Relatedly, documentation around data products is more important than traditional software products because there isn't really a tangible UI for data products.
6. Data discovery platforms are very useful for data consumers because they are not only for discovering that data products exist but they are also great for discovering information about the data products - they provide a good initial understanding of data products before digging deeper.
7. In data mesh, you shouldn't think of one single data platform to rule over everything. Users care about capabilities and their workflows/UX, not about exactly how things are stitched together on the back-end. Don't have 100 platforms but don't also create a monolithic beast you can't evolve.
8. ?Controversial?: Product discovery - the act of discovering what product(s) you should build - is crucial and often overlooked in the data product space. Many organizations are waiting for requests instead of data producers discovering what data products users might want.
9. While we should contextualize product management functions and needs to data product management, oftentimes the terminology gets all jumbled up. Be very explicit in defining terms to other stakeholders to prevent as much confusion as possible - but there will still be confusion :)
10. To help guide you in creating the right data product(s) to serve a use case, you really need to work closely with consumers - what are the day-to-day business problems of the use case? Then you can start to work backwards towards creating the right data products to serve the use case.
11. Make sure to dig into who will actually use a data product and how will it drive value for them. There are far too many business processes that are wildly inefficient or even unnecessary simply because people aren't asking - or asking again - those questions. Don't let your data product be inefficient, be intentional in asking.
12. Leverage the 'Product Trio' concept for data product management. While you won't have a design lead, find someone that can still help to optimize the data user experience (DUX) instead of the UI as you would with software. Again that UX is data fluency, access, and documentation.
13. Use product management frameworks to deep dive into potential data users' actual challenges. Get specific so you can find points of leverage to improve the process. To actually drive value, you have to drill down.
14. A 'fast fail' culture/approach is crucial to getting the best value from data. Producers and consumers can quickly test what works and iterate towards better instead of making data work all or nothing. Failure in data has to be something we are okay with.
15. Potentially the hardest aspect of data product management is defining success and the metrics to measure your success. Don't feel bad if you are struggling here too :) It's okay to start with less than great metrics and iterate towards better.
16. Enabling self-serve for consumers of data products should fall on the platform rather than on the data product owner. If the platform isn't robust enough, you shouldn't make every data product manager build out those capabilities to make up for it.

Frannie started off with a bit about her background and how that has helped her adopt the data as a product mindset, applying product thinking, when looking at creating and maintaining data products. A lot of it is about mapping traditional software product management to data product management as many things translate pretty well 1:1 but lot of things _don't_. And knowing the difference is a crucial point of leverage. The data product isn't the point, it's merely a way to exchange information to support a use case. In software product management, the product manager "serves as a bridge between software engineering and software users". In data product management, the product manager should serve as the bridge between data producers and consumers to ensure consumer requirements are satisfied.

Martin Eriksson's idea that product management is the intersection of tech, business, and user experience is foundational to Frannie. However, what actually is a user experience (UX) relative to a data product? Much of the user interface (UI) is actually owned by the platform itself rather than the data product so that makes UX a little more nebulous. She believes we should break down the UX into three categories: data fluency, access, and documentation. Access is the easiest because it's about making sure people can request access and can grant access easily - yes, automatic access is great too but that's not always possible. Data fluency is about how well the data product communicates its information to target audiences. Can people understand what the data product contains, why it exists, how things are organized, etc.? Lastly, unfortunately it seems documentation is harder and also more important in data product management than software product management. In software there is a tangible user interface but that's not really feasible for data products so the documentation is the way to guide people to understanding.

For Frannie, data discovery platforms are really valuable for consumers to understand what data products exist. And once they find data products that might be of interest, the platform allows them to learn more about the data product. Companies should make it easy to find the most useful information about a data product all in one place, e.g. the lineage, the purpose of the data product, sample queries, sample data, etc. as well as request access. Scott note: this is a partly a trapped metadata challenge. Many tools trap their metadata so it's hard to bring in things like SLA observability directly into the data discovery platform.

There is a big difference between data discovery and product discovery - or in this case data product discovery - in Frannie's terminology. In software product management, product discovery is about finding the needs users have and looking to fill that with product{s). Basically, discovering what products or features of products should exist. This is often overlooked in data product management because so much of the genesis for data products is defined use cases where the consumers come to the producers with their use case at least somewhat well scoped and understood. But data product discovery is a very useful practice to at least investigate if not lean more heavily into. Scott note: I've been saying this for literally over a year - communicate and spark the art of the possible with data consumers!

Frannie talked more about the process of discovering the right data product(s) to create. You want to dig in deep with the potential users. They might have ideas about what data they want but the most important aspect is what is the day-to-day challenge or opportunity they are looking to address? What is the actual business process and how do they want it improved? That will help tell you what data product(s) you need to create for the use case and how you need to shape it to address the business process they want to make better. Essentially, consumers are experts in what they are trying to fix, don't rely on them to be experts in what data to share and how, that's for the producers to own.

The 'Product Trio' concept from Teresa Torres has really been helpful to Frannie and team around data product management. While it is coming from the product management space where the trio is the product manager, the technical lead, and the product designer, it needs to be altered for data products since that designer role doesn't make as much sense when there isn't a direct UI. So you still have a product manager, a tech lead of some kind, and then someone that is more aligned to the user, maybe a data scientist or a data architect, to help ensure a positive data user experience (DUX). Again, Frannie breaks that DUX down into 3 components: data fluency, access, and documentation.

Frannie recommends looking at product management frameworks for leading product discovery sessions, e.g. the value proposition framework. Frameworks can help you get to actual tangible information about business processes and their challenges. Think about concrete ways to qualify and potentially even quantify aspects of the business challenges. Get pretty specific instead of a laundry list of potential use cases, drill into one or two and figure out how to help. Scott note: this seems to be a recurrent issue in data - lack of specificity around use cases and needs. It's part of why requirements gathering fails. Do not skimp on this process :D

Failing fast is a concept Frannie believes we need to adopt - and maybe adapt - in data. That's what prototyping needs to be better able to quickly get to value. Failure has historically been somewhat of a catastrophe in data because so many things have been essentially all or nothing with huge budgets. But with things like data mesh and in general with collaborative iterative data work between producers and consumers, failure doesn't mean forever fail, it leads us to places to improve. So while we might need new terminology around fast fail - data people seem to hate the concept of failing at all - it's a practice that will help us quickly iterate towards better solutions and more value. A fast fail culture with smaller blast radii can really highlight how people will use a data product instead of the pre-planning phase where you try to come to how both sides _think_ a consumer will use a data product.

Frannie shared what probably all data product managers feel but few hear: it's very difficult - and probably a bit squishy - to define success for a data product and then also define the metrics to measure success. E.g. just pointing to number of users doesn't necessarily equate to value. User satisfaction is a useful measure but obviously it doesn't cover every aspect. How well are you doing at hitting your SLAs? It's okay to start with non-perfect metrics as long as they are directionally valuable. Scott note: I recommend listening to episode #95 with Dave Colls on fitness functions if you want to dig deeper into success metric tracking.

Quick tidbit:

It's important to build out your data consumer self-serve capabilities to give consumers a better feeling of being able to actually get necessary access. And it's even more important to make sure that is handled by the platform and not the individual data products.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf