#43 - UX Benchmarking: Demonstrate Design ROI with Kate Moran of NN/g

May 29, 2020

Kate Moran, VP of Research & Content at Nielsen Norman Group, dives into the world of UX benchmarking. She simplifies the process of demonstrating design ROI and highlights its importance for getting stakeholder buy-in. Kate discusses how to choose the right metrics and differentiate between summative and formative studies. She shares real-world case studies, effective strategies for gauging task efficiency, and addresses audience questions on minimizing bias in research. A lively exploration of UX's quantitative side that combines learning with fun!

Ask episode

AI Snips

Chapters

Books

Transcript

Episode notes

INSIGHT

Core Concept of UX Benchmarking

UX benchmarking compares quantitative UX metrics against meaningful reference points for assessing user experience.
Without a valid comparison, numbers alone can't reveal if performance is good or bad.

ADVICE

Choosing Right Metrics Matters

Choose benchmarking metrics aligned with your product's core purpose and business goals.
Pick metrics meaningful long-term, not just tied to trendy features or short-term excitement.

INSIGHT

Formative vs. Summative Studies

Formative studies help shape design, providing qualitative insights during development.
Summative studies, like "final exams," evaluate overall design effectiveness at defined points.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

UX benchmarking may seem like a lot of work, but Kate Moran is here to show you how to do it effectively. She's VP of Research & Content at Neilsen Norman Group and leads UX teams to better benchmarking, teaches newbies how to get started, and explains this complicated subject with clarity. She joined Erin and JH on our very first live episode to explain how UX benchmarking can help teams show the ROI of their work.

She walked through how benchmarking can help get stakeholders on board, how to choose the right metrics early on, and most importantly, how to translate that to real ROI.

Our very first live podcast was a great learning experience and a ton of fun! We really enjoyed the interactive aspect, and our audience asked a lot of thoughtful questions.

Highlights

[2:01] Kate explains what UX benchmarking is
[3:37] How to choose benchmarking metrics
[12:01] The difference between summative and formative studies, and why you need to distinguish between them.
[17:21] Why context matters when evaluating benchmarking metrics
[21:28] How to translate benchmarking results to ROI
[29:11] Kate talks about case studies from NNg's ROI for Usability report
[35:34] Q&A - How do you limit bias in unmoderated studies with non-users and users?
[38:16] Q&A - How do you measure time spent on a task? Stopwatches aren't great.
[39:59] Q&A - How do session replay tools fit into this?
[41:10] Q&A - What happens when your stakeholders have different metrics for success?
[44:57] Q&A - If a participant thinks they completed a task successfully, is that a success?
[46:42] Q&A - How do you benchmark for emotional aspects, like how fun a product is?
[49:07] Parting words of wisdom

Kate's recommended resources

NN/g full-day seminar: Measuring UX & ROI
NN/g articles about benchmarking, ROI, and quantitative research
Jeff Sauro’s articles about quantitative UX research (he also has very useful confidence interval and statistical significance calculators): measuringu.com
Jeff Sauro’s book about benchmarking: Benchmarking the User Experience
Two great books for getting started with statistics: Straightforward Statistics and The Cartoon Guide to Statistics