

#166 - Summer Throwback: Erika Hall on Why Surveys (Almost Always) Suck
May 15, 2025
This week, Erika Hall, founder of Mule Design and author of 'Just Enough Research,' dives into the pitfalls of surveys in research. She argues that surveys are often misused, yielding unreliable data and that teams should focus on more meaningful conversation instead. Erika highlights the dangers of quick, poorly designed surveys and the importance of starting with the right research questions. Instead of seeking predictions, she suggests asking about past behaviors and prioritizing quality insights over quantity.
AI Snips
Chapters
Books
Transcript
Episode notes
Surveys Are Dangerously Misused
- Surveys are the most dangerous and misused research tool because they're easy to run but hard to design correctly.
- Ease of online surveying creates overconfidence and poor decisions based on flawed evidence.
Big Samples Can Mislead
- Large sample sizes feel authoritative but can mask invalid questions and biased samples.
- Numbers alone don't guarantee meaningful or accurate insight about behavior.
Start With The Question Not The Method
- Start by defining the higher-order question: what do we actually need to know to inform a decision?
- Choose methods that can validly answer that question rather than defaulting to a survey.