STEM-Talk Episode 77: John Ioannidis discusses why most published research findings are false
Our guest today is Dr. John Ioannidis, a Stanford professor who has been described by “BMJ” as “the scourge of sloppy science.” Atlantic magazine has gone so far as to refer to him as one of the world’s most influential scientists.
John is renowned for his 2005 paper, “Why Most Published Research Findings Are False,” which has been viewed more than 2.5 million times and is the most citied article in the history of the journal PLoS Medicine. He has authored nearly a thousand academic papers and has served on the editorial board of 30 top-tier journals.
At Stanford, John is a professor of medicine, of health research and policy, and of biomedical data science in the school of medicine as well as a professor of statistics in the school of humanities and sciences. He is the co-director of the university’s Meta-Research Innovation Center and the former director of the Stanford Prevention Research Center.
In today’s wide-ranging interview, John talks about:
- [00:07:43] What led him to begin questioning the reliability of medical research during his residency at Harvard.
- [00:12:03] His 2005 paper, “Why Most Published Research Findings Are False.”
- [00:26:27] How a major issue facing science is a lack of replication.
- [00:30:51] Which studies are worse, nutritional studies or drug studies.
- [00:38:25] If it’s possible to remove sampling biases like the healthy user bias.
- [00:46:50] The need for scientists to disclose their personal dietary biases as well as their personal diets when publishing research findings.
- [00:52:40] His recent paper, “Evidence Based Medicine Has Been Hijacked,” which argues that vested interests have transformed clinical medicine into something that resembles finance-based medicine.
- [00:55:36] The impact that funding pressure is having on the veracity of research being done today.
- [01:08:42] The need for future research to be designed by scientists without vested interests.
- [01:14:58] The ways John would fix the system if he had magic wand.
- [01:18:42] And as a bonus, John reads an excerpt from his latest book.
Show notes:
[00:02:37] Dawn begins the interview asking John about being born in New York but raised in Athens.
[00:03:54] John talks about how his parents were physicians and researchers and how they instilled in him a love for mathematics at an early age.
[00:05:26] Dawn asks John about winning the Greek Mathematical Society’s national award when he was 19 years old.
[00:06:23] John talks about his decision to go to medical school and to attend Harvard.
[00:07:43] Ken mentions that John began questioning the reliability of medical school during his residency at Harvard, and asks John to talk about his interest in an “evidence-based medicine” movement that was gathering momentum at the time.
[00:08:47] Dawn asks John about his work with the late Tom Chalmers, who played a major role in the development of randomized controlled trials.
[00:09:58] John talks about returning to Greece to take a position at the University of Ioannina.
[00:12:03] John talks about his 2005 paper “Why Most Published Research Findings Are False,” which became the single most-cited and downloaded paper in the history of the journal PLoS Medicine.
[00:15:32] Dawn mentions that when the paper came out, it was theoretical model. She asks John to talk about how now there are a number of studies pointing out problems with preclinical research on drug targets.
[00:17:34] Dawn asks John about his decision to leave the University of Ioannina to take a position at Stanford University.
[00:21:02] Dawn asks John for his thoughts on ways to improve the peer-review process.
[00:24:09] John talks about how he and his colleagues have found that most medical information that doctors rely on is flawed.
[00:26:27] Dawn points out that a major issue facing science is a lack of replication. She talks about how funding for repeat studies is hard to come by and that many journals will reject data that replicates previously published work. She asks John for his thoughts about ways to change this.
[00:29:14] Ken asks John if he sees interesting variations across disciplines.
[00:30:51] Ken follows up with a question about which studies are typically worse: nutritional studies or drug studies?
[00:31:28] John talks about the issues with nutritional epidemiology that stems from the often-implausible results arising from food frequency questionnaires.
[00:33:04] Dawn mentions that John has talked about the importance of getting nutritional epidemiology right because people are dying from bad decisions. She asks John if we’re at a point where we should scrap all nutritional epidemiological findings.
[00:36:20] Ken talks about how John has shown that almost all of the variables in nutritional epidemiology are connected, which partly explains multiple positive results. Ken asks John if there is a way to adjust for that.
[00:38:25] Ken points out that many studies proclaim they have adequately adjusted for multiple known health risk factors such as obesity, physical activity, and smoking. Ken asks John if it’s possible to do this in an accurate way in order to remove sampling biases like the healthy user bias.
[00:41:21] Dawn mentions that John in the past has said that a way to move forward in nutritional science is to run more large, long-term randomized trials. Considering that the problem is often a lack of behavior change due to the physical and social environment rather than a lack of knowledge about what to eat, she asks John what is the likelihood that large and expensive trials would truly give a meaningful result.
[00:44:49] Dawn asks John about the PREMIDED trial, which had originally been characterized as a rigorous and large randomized trial with long-term outcomes, but was later revealed not to be a randomized trial.
[00:46:50] Ken asks John about his argument that scientists should disclose their personal dietary biases as well as their personal diets when publishing their research findings.
[00:49:24] Dawn talks about the need for the scientific process to be objective, rigorous and ruthless about what is fact and what isn’t. She applauds John for his work that has shown how easy it is for researchers to manipulate data and results, and asks him to talk about the ways research is being manipulated.
[00:52:40] Dawn asks John to talk about his paper, “Evidence Based Medicine Has Been Hijacked,” which makes the argument that vested interests have transformed clinical medicine into something that resembles finance-based medicine.
[00:55:36]Dawn talks about how intense competition for funding has led to an aggressive environment where scientists are tempted to skew their findings to influence future funding. She asks John for his thoughts on the impact that funding is having on the veracity of research being done today.
[00:56:59] Ken asks John to elaborate on his comments that the healthcare system has become a threat to human health.
[00:58:39] John shares some ideas about ways to fix the healthcare system.
[00:59:39] Ken mentions an article John wrote that described how the biomedical research community has promoted the idea that genetics coupled with information technology will improve and transform healthcare and human health. Ken asks John to talk about why he thinks this approach has largely failed, and also talk about his proposal for a wholesale reevaluation of biomedical research.
[01:02:49] John talks about his research into “hyper-prolific” authors.
[01:06:13] Dawn brings up the need for universities to do a better job educating future scientists on ways to design studies and recognize biases. She asks John if he thinks we’re making progress in this area.
[01:08:42] Dawn points out that John in the past has talked about the entrenched bias in research and the need for future research to be designed by those without vested interests. She asks John if there’s a realistic way for researchers to balance expertise versus bias.
[01:10:39] Ken says it seems that the research enterprise is currently following a model of “product control,” that is, assessing the research once it has been finished and written up. He adds that this model coupled the reality of “publish or perish” thinking seems to be driving a lot of the problems we’re seeing with research quality and replication. Ken asks John how do we move from a “product control” model to more of a “process control” method in science, as is used in manufacturing, to ensure that the finished product is good by default.
[01:13:11] Dawn points out that there’s an increasing number of journals, including open access journals, and that this may lead to lower-quality research findings being published. She asks John for his thoughts.
[01:14:58] John talks about ways he would fix the system if he had magic wand to wave at the problem.
[01:17:29] Dawn points out that John has been writing literature since he was eight years old and has published seven books. She asks John about the role writing has played in his life.
[01:18:42] John ends the interview reading an excerpt from his latest book.
Links:
List of John’s published research
Atlantic article about Ioannidis
Why Most Published Research Findings Are False
Evidence-based Medicine Has Been Hijacked
What Happens When Underperforming Big Ideas in Research Become Entrenched