NN/G UX Podcast 56. AI for UX Analysis: How Accurate Is It? (feat. Christian Holst & Jamie Holst, Baymard Institute)
Dec 19, 2025
Christian Holst, Co-founder and Research Director at Baymard Institute, and Jamie Holst, Co-founder and CTO, delve into the reliability of AI tools in UX analysis. They stress the crucial role of accuracy, revealing that even 70% accuracy can lead to harmful recommendations. The discussion highlights the risks faced by junior UX practitioners over-relying on AI, as well as the importance of data-driven decisions in user experience. The duo also emphasizes the need for transparency in AI tool performance and the significance of owning the user experience outcomes.
AI Snips
Chapters
Transcript
Episode notes
Why Baymard Was Founded
- Jamie and Christian started Baymard because design decisions were often made by opinion rather than data.
- They wanted measurable, research-backed guidance for e-commerce product pages and conversion decisions.
Accuracy Trumps Hype
- Christian found ChatGPT-4 achieved only ~20% accuracy on heuristic audits versus human experts.
- Low accuracy tools produce harmful recommendations that can negate real improvements.
Require Documented Accuracy
- Ask AI vendors to publish documented accuracy rates before using their tools for product decisions.
- Demand the documentation so you can judge whether the tool's evaluation set is representative of your site.