Science Fictions Episode 93: Many analysts
19 snips
Jan 13, 2026 Discover the intriguing Many-Analysts Problem, where identical datasets yield wildly different conclusions. Delve into examples from sports data highlighting bias, and explore how researchers' choices shape findings in studies. The discussion spans fMRI results, ideological influences, and the impact of data cleaning on outcomes. With critiques of traditional analysis and suggestions for improvement, this episode invites listeners to rethink the objectivity of science. Join the conversation on transparency, collaboration, and what it means for research credibility!
AI Snips
Chapters
Books
Transcript
Episode notes
Many Analysts, Many Valid Paths
- Different analysts often choose very different reasonable methods when given the same dataset.
- Those choices can lead to widely varying conclusions even without deliberate p-hacking.
Football Data And 29 Research Teams
- The original 'Many Analysts' study gave 29 teams football data on skin tone and red cards.
- Teams produced a wide spread of results about referee bias, from null to strong effects.
Choices Drive Divergence More Than Malice
- Researcher degrees of freedom produce the Garden of Forking Paths effect.
- Even reasonable, single-step choices can lead to divergent published outcomes.
