Stuart Ritchie, "Science Fictions: Exposing Fraud, Bias, Negligence, and Hype in Science" (Penguin Books, 2020)
Aug 10, 2020
auto_awesome
Stuart Ritchie, a professor of psychology, discusses how the scientific enterprise falls short of its truth-seeking ideals due to systemic issues. He exposes unreliable, exaggerated, and fraudulent papers that influence widely accepted theories and claims. Ritchie highlights the biases introduced by even well-meaning scientists and the inadequate training they receive. The discussion covers the replication crisis, data manipulation, media hype, unconscious priming, the issue of replication, statistical analysis, and the role of open science in combating fraud and bias in the scientific community.
The replication crisis in science highlights the unreliability of many widely accepted theories and claims.
Biases, questionable statistical methods, and selective reporting contribute to the production of untrustworthy scientific results.
Flaws in the scientific process, from grant applications to publication, lead to a distorted scientific literature that fails to reflect the full scope of research conducted.
Deep dives
The Replication Crisis in Science
The podcast discusses the replication crisis in science, which emerged in psychology in 2012. It was sparked by prominent papers that could not be replicated by independent groups. Examples include the priming research in social psychology. This crisis led to large-scale replication attempts in various fields, revealing the unreliability of many studies. The scientific literature is filled with biases, questionable statistical methods, and selective reporting, undermining the foundation of knowledge.
Scientific Misconduct and Untrustworthy Results
The podcast highlights how scientists can inadvertently produce untrustworthy results. Factors include sloppily written papers that fail to describe the experiment accurately, cases of fraud or falsified results, and biased statistics. Scientists may manipulate analyses by dropping participants, adding covariates, or running numerous tests until a statistically significant result is obtained. The pressure to publish positive results and the misconception that statistical significance equates to truth contribute to the problem.
The Flaws in the Scientific Process
The podcast explores the flaws in the scientific process, from grant applications to publication. Scientists face challenges in securing funding, often hyping their research to stand out. Publishing bias leads to the file drawer effect, where non-significant results go unreported. Journals favor significant results, perpetuating the problem. Peer review, a critical gatekeeping process, is prone to biases and lacks training. The cumulative effect is a distorted scientific literature that fails to reflect the full scope of research conducted.
The Issues with P-values and Statistical Analysis
The podcast explains the issues with p-values and statistical analysis. The misuse of p-values contributes to p-hacking, where researchers manipulate analyses to achieve significant results. HARKing (Hypothesizing After the Results are Known) is another problem, where researchers change the hypotheses after analyzing the data to fit the significant findings. The arbitrary threshold of p<0.05 as a significant result leads to biased publication decisions and unreliable scientific conclusions. The complexity of statistics and a lack of proper training further exacerbate these issues in scientific research.
The Importance of Preregistration in Science
Preregistration is a process borrowed from medical trials where researchers plan and declare their analysis methods before conducting the research. This ensures transparency and reduces the likelihood of HARKing (hypothesizing after results are known) and p-hacking. Preregistration has been proven effective in preventing publication bias and increasing the reporting of null results. By publicly posting analysis plans, researchers are held accountable for their methodology, while still allowing for serendipitous discoveries through exploratory analysis.
The Role of Open Science in Improving Research Practices
The open science movement aims to increase transparency in the scientific process by promoting open access, open data, and preprints. Open access ensures that scientific papers are freely accessible to everyone, reducing financial barriers and allowing for wider dissemination of knowledge. Open data and preprints further enhance transparency by making raw data, code, and early versions of research papers available for scrutiny. These changes, alongside improved peer review practices and better publication methods, contribute to more rigorous science and help address issues of fraud, bias, negligence, and hype.
So much relies on science. But what if science itself can’t be relied on? In Science Fictions: Exposing Fraud, Bias, Negligence, and Hype in Science (Penguin Books, 2020), Stuart Ritchie, a professor of psychology at King’s College London, lucidly explains how science works, and exposes the systemic issues that prevent the scientific enterprise from living up to its truth-seeking ideals.
While the scientific method will always be our best way of knowing about the world, the current system of funding and publishing incentivizes bad behavior on the part of scientists. As a result, many widely accepted and highly influential theories and claims—priming, sleep and nutrition, genes and the microbiome, and a host of drugs, allergies, and therapies—are based on unreliable, exaggerated and even fraudulent papers. Bad incentives in science have influenced everything from austerity economics to the anti-vaccination movement, and occasionally count the cost of them in human lives.
Stuart Ritchie has been at the vanguard of a movement within science aimed at exposing and fixing these problems. In this New Books Network conversation, we speak specifically about how even the most well-meaning and truth-seeking scientists can unwittingly introduce bias into their analyses. We discuss ways that scientists’ training is inadequate.
Matthew Jordan is a professor at McMaster University, where he teaches courses on AI and the history of science. You can follow him on Twitter @mattyj612 or his website matthewleejordan.com.
