Experiencing Data w/ Brian T. O’Neill (UX for AI Data Products, SAAS Analytics, Data Product Management)

This is part two of the framework; if you missed part one, head to episode 175 and start there so you're all caught up.  In this episode of Experiencing Data, I continue my deep dive into the MIRRR UX Framework for designing trustworthy agentic AI applications. Building on Part 1’s “Monitor” and “Interrupt,” I unpack the three R’s: Redirect, Rerun, and Rollback—and share practical strategies for data product managers and leaders tasked with creating AI systems people will actually trust and use. I explain human-centered approaches to thinking about automation and how to handle unexpected outcomes in agentic AI applications without losing user confidence. I am hoping this control framework will help you get more value out of your data while simultaneously creating value for the human stakeholders, users, and customers. Highlights / Skip to: Introducing the MIRRR UX Framework (1:08) Designing for trust and user adoption plus perspectives you should be including when designing systems. (2:31) Monitor and interrupt controls let humans pause anything from a single AI task to the entire agent (3:17) Explaining “redirection” in the example context of use cases for claims adjusters working on insurance claims—so adjusters (users) can focus on important decisions. (4:35)  Rerun controls: lets humans redo an angentic task after unexpected results, preventing errors and building trust in early AI rollouts (11:12) Rerun vs. Redirect: what the difference is in the context of AI, using additional use cases from the insurance claim processing domain  (12:07) Empathy and user experience in AI adoption, and how the most useful insights come from directly observing users—not from analytics (18:28) Thinking about agentic AI as glue for existing applications and workflows, or as a worker  (27:35) Quotes from Today’s Episode The value of AI isn’t just about technical capability, it’s based in large part on whether the end-users will actually trust and adopt it. If we don’t design for trust from the start, even the most advanced AI can fail to deliver value." "In agentic AI, knowing when to automate is just as important as knowing what to automate. Smart product and design decisions mean sometimes holding back on full automation until the people, processes, and culture are ready for it." "Sometimes the most valuable thing you can do is slow down, create checkpoints, and give people a chance to course-correct before the work goes too far in the wrong direction."   "Reruns and rollbacks shouldn’t be seen as failures, they’re essential safety mechanisms that protect both the integrity of the work and the trust of the humans in the loop. They give people the confidence to keep using the system, even when mistakes happen." "You can’t measure trust in an AI system by counting logins or tracking clicks. True adoption comes from understanding the people using it, listening to them, observing their workflows, and learning what really builds or breaks their confidence."   "You’ll never learn the real reasons behind a team’s choices by only looking at analytics, you have to actually talk to them and watch them work."   "Labels matter, what you call a button or an action can shape how people interpret and trust what will happen when they click it." Quotes from Today’s Episode Part 1: The MIRRR UX Framework for Designing Trustworthy Agentic AI Applications 

Discover the innovative MIRRR UX framework designed to enhance trust in agentic AI applications. Learn about the necessity of human interaction and control surfaces in a world leaning towards automation. With engaging use cases from insurance claims processing, explore the first key points: Monitor and Interrupt, which focus on transparency and human oversight. Unpack the complexities of user interface design that balance speed and compliance, while addressing the critical need for incremental tasks to build trust amid AI decision-making.

In this episode of Experiencing Data, I chat with Irina Malkova who is the VP of AI Engineering and VP of Data and Analytics for Tech and Product at Salesforce. Irina shares how her teams are reinventing internal analytics, combining classic product data work with cutting-edge AI engineering—and her recent post on LinkedIn titled “AI adoption moves at the speed of user trust,” having a strong design-centered perspective, inspires today’s episode. (I even quoted her on this in a couple recent product design conference talks I gave!)  In today’s drop, Irina shares how they’re enabling analytical insights at Salesforce via a Slack-based AI agent, how they have changed their AI and engineering org structures (and why), the bad advice they got on organizing their data product teams, and more. This is a great episode for senior data product and AI executives managing complex orgs and technology environments who want to see how Salesforce is scaling AI for smarter, faster decisions.

Todd Olson, CEO of Pendo, dives into the evolving landscape of analytics SAAS and how to make data worth paying for. He discusses overcoming analytics apathy by simplifying dashboards and ensuring insights are actionable. Olson emphasizes the importance of 'time to value' and user adoption over mere engagement metrics. He also explores the future of AI product management and the unique hiring practices at Pendo, focusing on analytical skills over traditional backgrounds. Concerns about AI-generated answers and user trust further enrich this insightful conversation.

Tony Zhang, a researcher at Fortis with expertise in human-computer interaction, shares fascinating insights on AI design and user interaction. He discusses the risks of automation blindness, particularly in aviation where over-reliance on AI can hinder critical thinking. Zhang presents research showing how forward reasoning improves decision-making compared to traditional recommendations, emphasizing a hybrid approach that combines AI guidance with human intuition. The conversation also touches on enhancing user trust and the need for thoughtful AI integration in investment decisions.

Explore the pathways into data and AI product management, focusing on essential skills and mindsets for success. Discover the critical distinctions between product and project management, and why soft skills are crucial for navigating user adoption. Learn how value creation supersedes mere metric tracking, and why a business-oriented approach paired with hybrid skills is vital for fostering user engagement and driving financial success.

Today, I'm chatting with Shri Santhanam, the  EVP of Software Platforms and Chief AI Officer of Experian North America. Over the course of this promoted episode, you’re going to hear us talk about what it takes to build useful consumer and B2B AI products. Shri explains how Experian structures their AI product teams, the company’s approach prioritizing its initiatives, and what it takes to get their AI solutions out the door. We also get into the nuances of building trust with probabilistic AI tools and the absolutely critical role of UX in end user adoption.   Note: today’s episode is one of my rare Promoted Episodes. Please help support the show by visiting Experian’s links below:     Links Shri's LinkedIn Experian Assistant | Experian Experian Ascend Platform™ | Experian 

Stuart Winter-Tear, Chief Product Officer at Altima's Ventures, shares his extensive expertise in AI product management. He discusses the critical role of user experience in LLM-powered products and argues that many innovations stem from FOMO rather than genuine user needs. Stuart emphasizes the importance of crafting value-driven solutions that prioritize problem-solving over technology. He also explores the complexities of achieving product-market fit and the challenges of building user trust in AI products, highlighting the need for skilled 'translators' between tech and business.

Today, I am going to share some of the biggest challenges internal enterprise data leaders may face when creating their first revenue-generating data product. If your team is thinking about directly monetizing a data product and bringing a piece of software to life as something customers actually pay for, this episode is for you. As a companion to this episode, you can read my original article on this topic here once you finish listening!

Natalia Andreyeva, Senior Director of Product Management for the Nexus AI/ML Solution Portfolio at Infor, shares insights on AI product management for supply chain software. She emphasizes the critical role of user experience in B2B data products, advocating for strong customer engagement throughout the development process. The conversation delves into designing for AI, the importance of user discovery, and how to create compelling, trustworthy AI solutions. Natalia also addresses the nuances of integrating emerging technologies and the need for clear communication in user interfaces.