Peer Review, a Tarnished “Gold Standard”

Written by Richard Phelps on July 31, 2021. Posted in Current News

I recently submitted a manuscript to an education journal, a review essay of another scholar’s work. It opened with a compliment of the author’s “highly-praised and influential work.” To that statement, one reviewer of my manuscript asserted that I used “emotionally loaded language of incredulity, dismissiveness, and hyperbole.”

More “tone policing” comments riddled the review, suggesting that even when my words might sound benign or complimentary, what I really meant was malevolent. There were several examples of another curious critique type as well: Raw declarations that my claims could not possibly be true, without any effort having been made to follow my footnotes to the evidence.

Few of my manuscript submissions to education journals have been reviewed substantively. For those familiar with Paul Graham’s hierarchy of disagreement, the reviews tend to lack a refutation of the central point, refutation in general, or counterargument. Instead, most consist of responding to (perceived) tone or being ad hominem (even anonymously) attacks.

I consider it likely that this reviewer knew that I had written the manuscript. And that may have motivated the decision to volunteer. If I read the editor’s reviewer numbering system correctly, three others had agreed to review but then failed to follow through. Who knows how many were originally asked?

As it is, the traditional peer review system relies on unpaid volunteers from a small population of very busy people to perform an intensive and time-consuming task. Two types of scholars feel most compelled to review papers: Those intrinsically motivated (not necessarily for noble reasons) such as the aforementioned reviewer, and those who “have to,” such as graduate students and not-yet-tenured professors.

Assume an editor with a new manuscript chooses wisely and picks two eminently qualified reviewers, they accept the invitations, and then review the manuscript responsibly. Nonetheless, they are still just two people in an intellectual circle of thousands.

The odds are small that their consideration of the manuscript will bring to bear all the relevant knowledge or even a representative sample of it. I have written elsewhere about the widespread plague of “dismissive literature reviews,” time-saving and self-aggrandizing claims made by manuscript authors that no previous research exists on their topic. Those reviewing manuscript submissions generally lack the time to thoroughly review the new research presented, much less verify claims about entire research literatures.

Scholars have tabled many proposals to tweak the current system. Two widely discussed changes are to pay the reviewers, or open the process by eliminating anonymity in reviews (and thus discouraging the self-dealing and cruelty toward authors it now makes possible).

But these proposals maintain the traditional set-up: submit a manuscript, get it reviewed, then it’s rejected or published. If rejected, the same hit-or-miss process starts at another journal. The traditional review process can take months, or even years.

University and faculty raters construct their metrics in part from peer review results. Gaming these metrics corrupts the process with fake co-authors or reviews, citation cartels, co-citation agreements between journals or universities, reviewer- or editor-coerced citations, or purchased citations or by-lines.

Some published articles may later be retracted, but they represent a small proportion of those that probably should be, and even they are largely limited to the hard sciences. Retractions in the social sciences and humanities are far less common and, in education, less common still.

Given the well-deserved reputation of inferior quality in education research, its lower rate of retraction is more worrisome than reassuring.

One of the more intriguing solutions to peer review’s limitations and other faults would be to publish and then peer review.

With “post-publication review,” a submitted manuscript that meets the many other standards for publication (e.g., readable, understandable, relevant), is posted online as a temporary preprint and then critiques are welcomed via a broadcasted invitation to a wider circle of potential reviewers. (Of course, depending on how open the process is, some of the reviews might be irrelevant or unhelpful, just like the comments posted to many popular websites.)

PubPeer offers this process to a large extent and has already altered the fate of some articles. But it remains limited in popularity and seems to be most commonly used as an addendum to the traditional process. PubPeer’s allowance for anonymous reviews also engenders controversy.

Advantages of post-publication review include:

• Increasing the size of the pool of intrinsically motivated potential reviewers manyfold
• Reducing the reliance on overworked junior faculty, who may reallocate their time to reviewing articles of more interest
• Reducing the time and expense of the back-and-forth searching for and corresponding with potential reviewers.
• Getting new research findings out into the world more quickly
• Increasing process transparency
• Reducing opportunities for corruption among authors, editors, and reviewers
• Reducing opportunities for dismissive literature reviews (as those knowledgeable of prior relevant studies are provided the opportunity to contradict them publicly)

The traditional peer review process demands enormous amounts of time while providing only hit-or-miss and superficial appraisals of research quality. The result is a proliferation of scholarship of widely varying quality and much duplication of effort. This regime benefits scholars individually by thickening their curriculum vitae.

We may forget, however, that citation and publication numbers are not supposed to be ends in themselves but, rather, just proxies for adding useful and accurate information to society’s store of knowledge. Society might be better served by fewer scholarly journals publishing fewer articles that have been much more widely and thoroughly vetted.

See more here: jamesgmartin.center

About the author: Richard P. Phelps is founder of the Nonpartisan Education Group, editor of the Nonpartisan Education Review, a Fulbright Scholar, and a fellow of the Psychophysics Laboratory. He has authored, edited, and co-authored books on standardized testing, learning, and psychology.

Please Donate Below To Support Our Ongoing Work To Defend The Scientific Method

PRINCIPIA SCIENTIFIC INTERNATIONAL, legally registered in the UK as a company incorporated for charitable purposes. Head Office: 27 Old Gloucester Street, London WC1N 3AX. 

Trackback from your site.