Discover the challenges of quantifying the value of enterprise data products and the vital role of UX teams. Learn how value is subjective and extends beyond numbers. Explore the art of estimation, emphasizing qualitative assessments to communicate impact effectively. Delve into long-term innovation and ethics in measuring success, urging a shift from traditional ROI calculations. Engage with thoughtful insights on audience feedback and community involvement for continuous improvement.

In this discussion, Simon Landry, a Lead UX researcher at Thomson Reuters; Greg Nudelman, a distinguished designer at Sumo Logic; and Paz Perez, a UX designer at Google, explore the intricacies of designing user experiences for AI-driven products. They debate the challenges of ‘AI-first’ thinking, the importance of defining clear problems for LLMs, and the necessity for diverse design teams. The conversation emphasizes that while AI presents opportunities, it can also complicate user interactions, highlighting the critical need for user-centric approaches in AI design.

With GenAI and LLMs comes great potential to delight and damage customer relationships—both during the sale, and in the UI/UX. However, are B2B AI product teams actually producing real outcomes, on the business side and the UX side, such that customers find these products easy to buy, trustworthy and indispensable?    What is changing with customer problems as a result of LLM and GenAI technologies becoming more readily available to implement into B2B software? Anything?   Is your current product or feature development being driven by the fact you might be able to now solve it with AI? The “AI-first” team sounds like it’s cutting edge, but is that really determining what a customer will actually buy from you?    Today I want to talk to you about the interplay of GenAI, customer trust (both user and buyer trust), and the role of UX in products using probabilistic technology.     These thoughts are based on my own perceptions as a “user” of AI “solutions,” (quotes intentional!), conversations with prospects and clients at my company (Designing for Analytics), as well as the bright minds I mentor over at the MIT Sandbox innovation fund. I also wrote an article about this subject if you’d rather read an abridged version of my thoughts.   Highlights/ Skip to: AI and LLM-Powered Products Do Not Turn Customer Problems into “Now” and “Expensive” Problems (4:03) Trust and Transparency in the Sale and the Product UX: Handling LLM Hallucinations (Confabulations) and Designing for Model Interpretability (9:44) Selling AI Products to Customers Who Aren’t Users (13:28) How LLM Hallucinations and Model Interpretability Impact User Trust of Your Product (16:10) Probabilistic UIs and LLMs Don’t Negate the Need to Design for Outcomes (22:48) How AI Changes (or Doesn’t) Our Benchmark Use Cases and UX Outcomes (28:41) Closing Thoughts (32:36)   Quotes from Today’s Episode “Putting AI or GenAI into a product does not change the urgency or the depth of a particular customer problem; it just changes the solution space. Technology shifts in the last ten years have enabled founders to come up with all sorts of novel ways to leverage traditional machine learning, symbolic AI, and LLMs to create new products and disrupt established products; however, it would be foolish to ignore these developments as a product leader. All this technology does is change the possible solutions you can create. It does not change your customer situation, problem, or pain, either in the depth, or severity, or frequency. In fact, it might actually cause some new problems. I feel like most teams spend a lot more time living in the solution space than they do in the problem space. Fall in love with the problem and love that problem regardless of how the solution space may continue to change.” (4:51) “Narrowly targeted, specialized AI products are going to beat solutions trying to solve problems for multiple buyers and customers. If you’re building a narrow, specific product for a narrow, specific audience, one of the things you have on your side is a solution focused on a specific domain used by people who have specific domain experience. You may not need a trillion-parameter LLM to provide significant value to your customer. AI products that have a more specific focus and address a very narrow ICP I believe are more likely to succeed than those trying to serve too many use cases—especially when GenAI is being leveraged to deliver the value. I think this can be true even for platform products as well. Narrowing the audience you want to serve also narrows the scope of the product, which in turn should increase the value that you bring to that audience—in part because you probably will have fewer trust, usability, and utility problems resulting from trying to leverage a model for a wide range of use cases.” (17:18) “Probabilistic UIs and LLMs are going to create big problems for product teams, particularly if they lack a set of guiding benchmark use cases. I talk a lot about benchmark use cases as a core design principle and data-rich enterprise products. Why? Because a lot of B2B and enterprise products fall into the game of ‘adding more stuff over time.’ ‘Add it so you can sell it.’ As products and software companies begin to mature, you start having product owners and PMs attached to specific technologies or parts of a product. Figuring out how to improve the customer’s experience over time against the most critical problems and needs they have is a harder game to play than simply adding more stuff— especially if you have no benchmark use cases to hold you accountable. It’s hard to make the product indispensable if it’s trying to do 100 things for 100 people.“ (22:48) “Product is a hard game, and design and UX is by far not the only aspect of product that we need to get right. A lot of designers don’t understand this, and they think if they just nail design and UX, then everything else solves itself. The reason the design and experience part is hard is that it’s tied to behavior change– especially if you are ‘disrupting’ an industry, incumbent tool, application, or product. You are in the behavior-change game, and it’s really hard to get it right. But when you get it right, it can be really amazing and transformative.” (28:01) “If your AI product is trying to do a wide variety of things for a wide variety of personas, it’s going to be harder to determine appropriate benchmarks and UX outcomes to measure and design against. Given LLM hallucinations, the increased problem of trust, model drift problems, etc., your AI product has to actually innovate in a way that is both meaningful and observable to the customer. It doesn’t matter what your AI is trying to “fix.” If they can’t see what the benefit is to them personally, it doesn’t really matter if technically you’ve done something in a new and novel way. They’re just not going to care because that question of what’s in it for me is always sitting behind, in their brain, whether it’s stated out loud or not.” (29:32)   Links Designing for Analytics mailing list

Adam Berke, Chief Product Officer at The Predictive Index, shares his insights on merging company cultures and products after Charma's recent acquisition. He delves into the challenges of integrating two product teams and the vital need for clarity in leadership structures. Berke discusses how behavioral science shapes their hiring practices, enabling data-driven decisions. He also reflects on fostering employee-manager relationships and navigating the complexities of legacy customer expectations while pushing for innovation in a diverse workplace.

Andy Sutton, GM of Data and AI at Endeavour Group, shares his transformation from traditional analytics to a product-led approach in the liquor and hospitality sectors. He discusses the journey to build the 'Spotify for wines' through personalized data strategies that enhance customer experience. Andy highlights the importance of relationships in data-driven environments and the success his team achieved by focusing on user-centric design and agile methodologies. His insights into balancing analytics with real-world user needs illuminate the future of data products.

In this engaging discussion, Anna Jacobson, Operations and Data Partner at Operator Collective, shares her unique journey from construction management to data product design. She opens up about the art of persuading resistant users to embrace data products. Anna emphasizes the importance of user feedback and understanding diverse perspectives for successful product adoption. She also highlights how cultural challenges within organizations can impede progress and how engaging skeptics can turn them into advocates for data insights.

Ori Yudilevich, Chief Product Officer at MaterialsZone, discusses how his team harnesses machine learning and AI to revolutionize materials science R&D. He shares insights on creating a user-friendly SaaS platform that streamlines experimentation for scientists. The conversation highlights the importance of integrating product management, user experience, and data science to enhance efficiency. Ori also emphasizes the role of user feedback in refining designs and driving adoption, ultimately making materials testing faster and more cost-effective.

In this discussion, Jeremy Forman, VP of R&D, AI, Data, and Analytics at Pfizer, shares his vast experience in building AI-driven data products for pharmaceuticals. He explains the vital collaboration between data product analysts and UX designers in addressing user needs. Jeremy highlights the challenges of integration within medical data products and the importance of justifying UX budgets. He reflects on the transformative shift towards prioritizing user outcomes and how effective UX can enhance engagement and product adoption, ensuring that complex data solutions truly serve researchers.

Ovetta Sampson, Director of User Experience for Core Machine Learning at Google, is an expert in human-centered AI design. She discusses the delicate balance between AI innovation and ethical considerations. Topics include the critical roles of diverse teams in AI UX design, the urgent need for guardrails to protect data integrity, and the responsibilities of product teams as GenAI technology evolves. Ovetta also highlights the potential risks of mishandling AI and stresses the importance of thoughtful collaboration between humans and machines.

Founders of B2B SaaS analytics and AI startups often struggle with DIY UI/UX design, leading to confusion and stagnant progress. Common pitfalls include misjudging the balance between originality and replication, excessive customization, and the misunderstanding of key design concepts. There's a focus on the importance of clear value propositions and user-centric design. The discussion highlights when to seek outside expertise to enhance effectiveness and prevent customer attrition, ensuring a more compelling product experience.