#269 - Good vs. bad science: how to read and understand scientific studies
Sep 4, 2023
auto_awesome
In this discussion, Bob Kaplan, a collaborator on the 'Studying Studies' series, sheds light on how to navigate the complexities of scientific research. He and Peter dissect various study types, clinical trial phases, and biases that can distort findings. They emphasize the importance of differentiating relative and absolute risk and understanding statistical significance. Kaplan shares practical strategies for reading scientific papers critically and highlights the significance of rigorous study design, particularly in nutrition and drug trials.
Understanding the process of a study from idea to execution is crucial for evaluating scientific literature.
Different types of studies, including observational and experimental, have their own strengths and limitations.
Common biases in epidemiological studies, such as healthy user bias and recall bias, need to be recognized and mitigated.
Randomization, blinding, control groups, and study size are important factors for assessing the rigor and generalizability of a study.
Deep dives
Process of Conducting a Study
The podcast episode discusses the process that a study goes through, starting from the idea to the design and execution. It emphasizes the importance of having a hypothesis and formulating a null hypothesis. It explains the need for experimental design, determining sample size, obtaining ethical approval, and developing a plan for statistics. It highlights the significance of pre-registering the study and securing funding. The episode also mentions the various types of studies, including observational and experimental, and the importance of randomization in experimental studies.
Strengths and Limitations of Different Study Types
The podcast explores the strengths and limitations of different types of studies. It discusses the value of individual case reports in generating hypotheses and the importance of case series in studying rare phenomena. The episode explains the concept of cohort studies, both retrospective and prospective, and how they can provide valuable insights while controlling for certain variables. It also dives into the significance of randomized controlled trials as the gold standard and the benefits of blinding and control groups. The podcast briefly touches on meta-analyses and the need to critically evaluate the constitutive studies within them.
Common Biases in Epidemiological Studies
The podcast identifies and discusses common biases in epidemiological studies. It highlights the healthy user bias, where individuals who make certain lifestyle choices also engage in other health-promoting behaviors. It mentions recall bias and the challenges of relying on food frequency questionnaires for nutritional epidemiology. The episode also touches on performance bias, where the behavior or outcomes of participants in a study are influenced by the awareness of being observed or receiving special attention. Confounding variables are emphasized as potential sources of bias that need careful consideration and control in epidemiological studies.
Factors Influencing Study Confidence and Generalizability
The podcast delves into factors that can influence confidence in a study and its generalizability. It emphasizes the importance of randomization, blinding, and the presence of control groups in experimental studies. The episode also discusses the significance of study size, duration, and the population being studied. It highlights the trade-off between study heterogeneity and generalizability, as well as the need to consider multiple outcomes, distinguishing between primary and secondary outcomes, and addressing the issue of multiple hypothesis testing. Finally, it briefly mentions the challenges of blinding in certain types of studies, such as those involving psychedelics.
Importance of Pre-Registration and Peer Review
Pre-registration and peer review are key steps in the scientific research process. Pre-registration involves researchers submitting their study protocol, methods, and primary outcomes before conducting the study to ensure transparency and prevent bias. Peer review is the evaluation of the study by experts in the field to assess its quality, significance, and methodology. It helps ensure that studies are rigorous, accurate, and relevant before they are published.
Publication Bias and the Impact Factor
Publication bias is a concern in scientific research, where positive or significant results are more likely to be published than negative or inconclusive results. This bias can lead to an incomplete understanding of the research landscape. The impact factor of a journal is a measure of its prestige and influence in the scientific community. It is based on the ratio of citations to articles published in the journal. Journals with higher impact factors are generally more respected and have a greater influence in shaping scientific knowledge.
Review Process and Importance of Replication
The review process is an essential step in getting a study published in a journal. The manuscript is reviewed by experts who assess its quality, methodology, and significance. The reviewers can recommend acceptance, rejection, or revisions to the paper. The process aims to ensure that the study meets scientific standards and contributes to existing knowledge. Replication, or the repetition of studies by independent researchers, is crucial to confirm the validity and reliability of scientific findings. It helps identify errors, biases, or limitations in the original study and strengthens the scientific evidence base.
Reading a Scientific Paper
When reading a scientific paper, it is common to start with the abstract to determine if the study is of interest. If unfamiliar with the subject matter, reading the introduction can provide context. The methods section should be closely examined to understand study design, sample size, randomization, interventions, and measurements. Results are often presented in figures and tables, which should be carefully reviewed with corresponding legends. The discussion section provides interpretation, implications, and limitations of the study. The order of reading can vary based on familiarity and interests.
This special episode is a rebroadcast of AMA #30, now made available to everyone, in which Peter and Bob Kaplan dive deep into all things related to studying studies to help one sift through the noise to find the signal. They define various types of studies, how a study progresses from idea to execution, and how to identify study strengths and limitations. They explain how clinical trials work, as well as biases and common pitfalls to watch out for. They dig into key factors that contribute to the rigor (or lack thereof) of an experiment, and they discuss how to measure effect size, differentiate relative risk from absolute risk, and what it really means when a study is statistically significant. Finally, Peter lays out his personal process when reading through scientific papers.
We discuss:
The ever-changing landscape of scientific literature [2:30];
The process for a study to progress from idea to design to execution [5:00];
Various types of studies and how they differ [8:00];
The different phases of clinical trials [19:45];
Observational studies and the potential for bias [27:00];
Experimental studies: randomization, blinding, and other factors that make or break a study [44:30];
Power, p-values, and statistical significance [56:45];
Measuring effect size: relative risk vs. absolute risk, hazard ratios, and “number needed to treat” [1:08:15];
How to interpret confidence intervals [1:18:00];
Why a study might be stopped before its completion [1:24:00];
Why only a fraction of studies are ever published and how to combat publication bias [1:32:00];
Frequency of training for Olympic weightlifting [1:22:15];
How post-activation potentiation (and the opposite) can improve power training and speed training [1:24:30];
The Strongman competition: more breadth of movement, strength, and stamina [1:32:00];
Why certain journals are more respected than others [1:41:00];
Peter’s process when reading a scientific paper [1:44:15]; and
