Jamie Hale

Jamie Hale

Thursday, January 11, 2018

Peer Review is not the Antidote

 In the Peer Review Process a paper is submitted to a journal and evaluated by several reviewers (often reviewers are individuals with an impressive history of work in the area of interest-that is, the specific area that the article addresses).  After critiquing the paper the reviewers submit their thoughts to the editor.  Then, based on the commentaries from the reviewers, the editor decides
whether to publish the paper, make suggestions for additional changes that could lead to publication, or to reject the paper. 

Single Blind and Double Blind Reviews

In Single Blind Reviews authors do not know who the reviewers are.  In Double Blind Reviews authors do not know who the reviewers are, nor do reviewers know the identity of the authors.  In many fields Single Blind Reviews are the norm, while in others Double Blind Reviews are preferred.
“Peer review is one way (replication is another) science institutionalizes the attitudes of objectivity and public criticism.  Ideas and experimentation undergo a honing process in which they are submitted to other critical minds for evaluation.  Ideas that survive this critical process have begun to meet the criterion of public verifiability” (Stanovich, 2007, p. 12).

Criticisms Peer Review Process

Reviewers find it hard to remain Purely Objective due to their own education, experience and biases

The process is slow Critics point out, and this may deter submission of quality papers  

There are many examples of poor research published in peer-reviewed journals, which indicates the peer review process is often unsuccessful in preventing the publication of  bad science

Sometimes good research is not published, especially when findings are not statistically significant.  Publication bias is problematic and demonstrates confirmation bias

Reviewers are not always knowledgeable regarding contents they are reviewing (I have been asked to review papers that consist of contents that was outside my area of knowledge)

Lack of agreement and communication among reviewers

Reviewers tend to be highly critical of articles that contradict their own views, while being less critical of articles that support their personal views (example of myside bias). Well-known, established scientists are more likely to be recruited as reviewers

Lacks standardized criteria, and criteria for publication demonstrates huge variability among scholarly publications.

Final word-Peer Review Process
The Peer Review Process is not perfect, but some researchers suggest  it is one of the best safeguards we have against junk science (Stanovich, 2007). When evaluating the worth of scientific data, in addition to whether it is published in a peer reviewed journal, it is important to take into consideration:  funding sources, study replication, study design, sample size, conflicting interest, sampling error, different measures of reliability and validity, reporting limitations and other possible criticisms of study (special concern with *statistical validity- often not acknowledged or understood). 

There are good studies that never get published in peer review publications, and low quality studies are published by peer review publishers.  It is erroneous to label a study, review, commentary, meta-analysis or any other scholarly papers as high quality based solely on peer review status. This over glorification of peer review pervades academia and pop science.      

* When researchers question a study’s statistical validity they are questioning issues relevant to how well the conclusions coincide with the results, represented as statistics. Interrogating statistical validity may include some of the following questions: If the study found a difference what is the probability that the conclusion was a false alarm?  If the study’s finding found no difference what is the probability that a real relationship went unnoticed?  What is the effect size?  Is the difference between groups statistically significant? Are the finding practically significant? What type of inferential stats were used to assess predictions? Could different statistical procedures have been used? How would different samples influence statistical findings?  Stats make use of samples; inferences about the population are derived from data collected in the study.  It is important to avoid an exaggeration of the findings, consider sampling error, consider external validity and consider the need for converging evidence, and what the finding indicate regarding a representation of the population.      

References available upon request  

No comments:

Post a Comment