In this discussion, johnswentworth, the author behind "The Median Researcher Problem," dives deep into the nuances of scientific culture. He argues that it's the median researchers, rather than the top experts, who shape the prevailing ideas in a field, often leading to the spread of poor practices like p-hacking. He highlights the role of these median figures during the replication crisis, illustrating how their influence can obstruct improvement. The conversation offers a thought-provoking look at the collective impact of competence in research communities.
The Median Researcher Problem illustrates how the influence of median competence researchers, rather than top experts, shapes scientific norms and practices.
High-performing research communities can excel by prioritizing knowledge and critical thinking, yet often remain undervalued by the broader scientific landscape.
Deep dives
The Median Researcher Problem Explained
The concept of the Median Researcher Problem suggests that the influence of ideas within a scientific field is primarily shaped not by the top researchers but by those at the median level of competence. For instance, if most researchers lack a deep understanding of crucial methodologies like statistics and p-hacking, their flawed interpretations can become widely accepted despite their inaccuracies. This prevalence of poor practices can sustain faulty ideas even when more knowledgeable peers express concerns about them, as seen during the replication crisis. The inability of competent voices to change the prevailing narrative illustrates how the collective actions of median researchers dominate the memetic landscape of science.
Implications of Median Researchers on Scientific Communities
A significant implication of this issue is that small, high-performing research communities can effectively outpace larger fields due to stronger internal selection of ideas and practices. Such communities can foster a culture that prioritizes statistical knowledge and critical thinking, leading to more robust outcomes than the predominant practices within their broader fields. However, despite their superior performance, these groups may remain unrecognized or undervalued by the larger scientific community, as their higher-quality insights may not fit the prevailing norms. An example of this dynamic is observed in the LessWrong community, which benefits from an intelligent user base and a commitment to avoiding common errors, yet still faces challenges in being acknowledged as a thought leader.
1.
Exploring the Influence of Median Researchers in Science
Claim: memeticity in a scientific field is mostly determined, not by the most competent researchers in the field, but instead by roughly-median researchers. We’ll call this the “median researcher problem”.
Prototypical example: imagine a scientific field in which the large majority of practitioners have a very poor understanding of statistics, p-hacking, etc. Then lots of work in that field will be highly memetic despite trash statistics, blatant p-hacking, etc. Sure, the most competent people in the field may recognize the problems, but the median researchers don’t, and in aggregate it's mostly the median researchers who spread the memes.
(Defending that claim isn’t really the main focus of this post, but a couple pieces of legible evidence which are weakly in favor:
People did in fact try to sound the alarm about poor statistical practices well before the replication crisis, and yet practices did not change, so clearly at least [...]