The problems with peer reviewed research: Richard Smith

Peer review is at the heart of science. Yet, as Richard Smith explains in this talk, there are many problems with peer reviewed research. As Richard argues, peer review is not an evidence-based process, but rather a faith-based process. Is it time for something different? Peer review has two main functions: 1) Quality assurance and 2) Improving what is published. However, with that in mind, there is no evidence of effectiveness of peer review, and lots of evidence of its ineffectiveness. This is along with peer review being slow, expensive, wasteful, inefficient, prone to bias and being largely irrelevant! Richard discusses a few of the main issues as he sees it. For example, studies in large journals are more likely to be wrong when compared to smaller journals. Some argue that the vast majority of research is a waste of time. Similarly, replication is also a major problem, as Richard explains. A huge number of studies cannot be replicated, raising questions about the initial research. What is published in journals should not be taken to be truth but rather “provisional truth”. Richard tells the tale of planting errors in articles and seeing if they are picked up by peer-review. Far more often than not, they are not! On top of this, there is fraud and theft that occurs in the peer review process. Bias exists both positively and negatively. To top it off, much of the time the process does not pick up the errors (or fraud) in papers. A lot of problems and few answers. And it is not necessarily a case of it being the least “bad system” for this process anymore. There has been attempts to improve peer review. Blinding of reviewers to authors has been shown to be ineffective. An open system did not make much difference to the quality of opinion. Checklists and training have likewise been trialled and moved past due to ineffectiveness. Finally, the process has been made as open as possible, and in real time where possible. Richard argues that we may no longer need peer review. In the age of the internet, why would a peer review process be needed when an enormous number of people can access and critique the evidence simultaneously. Journals may be yesterday’s way of publishing data.

For more like this, head to https://codachange.org/podcasts/