From PNAS: What’s wrong with peer review

_{Denyse O'Leary

December 22, 2014

News, Peer review}

Share: Facebook; Twitter; LinkedIn; Flipboard; Print; Email

From a recent article’s Significance statement:

Our research suggests that evaluative strategies that increase the mean quality of published science may also increase the risk of rejecting unconventional or outstanding work.

You don’t mean stuff that is true but no elite source can afford to admit? Shocka! Abstract

Peer review is the main institution responsible for the evaluation and gestation of scientific research. Although peer review is widely seen as vital to scientific evaluation, anecdotal evidence abounds of gatekeeping mistakes in leading journals, such as rejecting seminal contributions or accepting mediocre submissions. Systematic evidence regarding the effectiveness—or lack thereof—of scientific gatekeeping is scant, largely because access to rejected manuscripts from journals is rarely available. Using a dataset of 1,008 manuscripts submitted to three elite medical journals, we show differences in citation outcomes for articles that received different appraisals from editors and peer reviewers. Among rejected articles, desk-rejected manuscripts, deemed as unworthy of peer review by editors, received fewer citations than those sent for peer review. Among both rejected and accepted articles, manuscripts with lower scores from peer reviewers received relatively fewer citations when they were eventually published. However, hindsight reveals numerous questionable gatekeeping decisions. Of the 808 eventually published articles in our dataset, our three focal journals rejected many highly cited manuscripts, including the 14 most popular; roughly the top 2 percent. Of those 14 articles, 12 were desk-rejected. This finding raises concerns regarding whether peer review is ill-suited to recognize and gestate the most impactful ideas and research. Despite this finding, results show that in our case studies, on the whole, there was value added in peer review. Editors and peer reviewers generally—but not always—made good decisions regarding the identification and promotion of quality in scientific manuscripts. (paywall)

As Retraction Watch explains, “Put another way, peer review rewards mediocrity at the expense of breakthroughs.” Hat tip: Retraction Watch Follow UD News at Twitter!

Comments

Peer review is the best we can do. Yes the system isn't perfect and certainly may need to be revamped, but it serves its purpose well. Don't worry guys, it isn't peer review that keeps ID out of credible journals; it's the lack of science behind ID that does this. Merry Christmas! Or happy Channukah, or kwanza or festivus!AVS_{December 22, 2014
December
12
Dec
22
22
2014
06:10 PM
6
06
10
PM
PDT}

Peer review promotes intellectual incest which inevitably leads to the birth of intellectual monstrosities. Examples of such monstrosities are: infinite number of parallel universes, time travel, immortality via brain uploading, dirt that can self-assemble into living organisms, conscious machines, intelligent machines that enslave or destroy humanity, legal rights for intelligent machines, etc.Mapou_{December 22, 2014
December
12
Dec
22
22
2014
05:24 PM
5
05
24
PM
PDT}

OT: Biological Information - Purifying Selection 12-20-2014 by Paul Giem - video https://www.youtube.com/watch?v=SGJZDsQG4kQbornagain77_{December 22, 2014
December
12
Dec
22
22
2014
04:54 PM
4
04
54
PM
PDT}

OT: The Case For Intelligent Design (What does the evidence say?) - Tom Woodward - video filmed Nov. 22, 2014 https://www.youtube.com/watch?v=5egamBlC4lMbornagain77_{December 22, 2014
December
12
Dec
22
22
2014
04:35 PM
4
04
35
PM
PDT}

Even as an egregiously unqualified layman, reading that sometimes outstanding work is turned down by a peer-review panel, only for its ideas to be stolen, plagiarized by one of the 'reviewers', made my hair stand on end.Axel_{December 22, 2014
December
12
Dec
22
22
2014
04:09 PM
4
04
09
PM
PDT}

The biggest problem with peer review is that it is inherently elitist. I don't think it is needed at all. Great thinkers of the past self-published their works. The entire world should be our peers, not just a minority of know-it-alls. I say, publish your stuff openly. If it is any good, it will survive and thrive. If not, it will disappear.Mapou_{December 22, 2014
December
12
Dec
22
22
2014
03:13 PM
3
03
13
PM
PDT}

a few notes:
An Interview with David Noble - Peer Review as Censorship by SUZAN MAZUR - 2010 Excerpt: SUZAN MAZUR: I’ve been focusing on abuse inside the peer review system in recent articles for CounterPunch. The system seems to have spiraled out of control – to the extent that at the low end we now find virtual death squads on Internet blogs out to destroy scientists who have novel theories. They pretend to be battling creationism but their real job is to censor the free flow of ideas on behalf of the science establishment. The science establishment rewards bloody deeds like these by putting the chief assassin on the cover of The Humanist magazine, for example. But you’ve written in "Regression on the Left" that the problem IS the peer review system itself. Why do you think so? David Noble: When you say THE problem is the peer review system – the peer review system in my view is doing what it was designed to do — censor. And filter. Peer review is a system of prior censorship, prior review – prior meaning prior to publication. So the idea of abusing the peer review system sort of adds insult to injury, because the peer review system itself is injurious.,,, http://www.counterpunch.org/2010/02/26/peer-review-as-censorship/ Interview with Suzan Mazur, the author of "The Altenberg 16: An Exposé of the Evolution Industry" (Corruption of peer-review by big science) - video http://www.ustream.tv/recorded/6515194 Everything we know is wrong, 26 August 2014 Excerpt: "Every day the newspapers carry stories of new scientific findings. There are 15 million scientists worldwide all trying to get their research published. But a disturbing fact appears if you look closely: as time goes by, many scientific findings seem to become less true than we thought. It's called the "decline effect" - and some findings even dwindle away to zero. A highly influential paper by Dr John Ioannidis at Stanford University called "Why most published research findings are false" argues that fewer than half of scientific papers can be believed,,,, He even showed that of the 49 most highly cited medical papers, only 34 had been retested and of them 41 per cent had been convincingly shown to be wrong. And yet they were still being cited. Again and again, researchers are finding the same things, whether it's with observational studies, or even the "gold standard" Randomised Controlled Studies, whether it's medicine or economics. Nobody bothers to try to replicate most studies, and when they do try, the majority of findings don't stack up. The awkward truth is that, taken as a whole, the scientific literature is full of falsehoods." http://www.bbc.co.uk/programmes/b04f9r4k The Folly of Scientism - Austin L. Hughes - Fall 2012 Excerpt: the high confidence in funding and peer-review panels should seem misplaced to anyone who has served on these panels and witnessed the extent to which preconceived notions, personal vendettas, and the like can torpedo even the best proposals. Moreover, simplistically defining science by its institutions is complicated by the ample history of scientific institutions that have been notoriously unreliable. http://www.thenewatlantis.com/publications/the-folly-of-scientism Nothing in biology makes sense except in light of theology? - Dilley S. - 2013 Abstract This essay analyzes Theodosius Dobzhansky's famous article, "Nothing in Biology Makes Sense Except in the Light of Evolution," in which he presents some of his best arguments for evolution. I contend that all of Dobzhansky's arguments hinge upon sectarian claims about God's nature, actions, purposes, or duties. Moreover, Dobzhansky's theology manifests several tensions, both in the epistemic justification of his theological claims and in their collective coherence. I note that other prominent biologists--such as Mayr, Dawkins, Eldredge, Ayala, de Beer, Futuyma, and Gould--also use theology-laden arguments. I recommend increased analysis of the justification, complexity, and coherence of this theology. http://www.ncbi.nlm.nih.gov/pubmed/23890740 "While the great majority of biologists would probably agree with Theodosius Dobzhansky’s dictum that “Nothing in biology makes sense except in the light of evolution”, most can conduct their work quite happily without particular reference to evolutionary ideas. Evolution would appear to be the indispensable unifying idea and, at the same time, a highly superflous one.” A.S. Wilkins, editor of the journal BioEssays, Introduction to "Evolutionary Processes" - (2000). "Biologists must constantly keep in mind that what they see was not designed, but rather evolved. It might be thought, therefore, that evolutionary arguments would play a large part in guiding biological research, but this is far from the case. It is difficult enough to study what is happening now. To figure out exactly what happened in evolution is even more difficult. Thus evolutionary achievements can be used as hints to suggest possible lines of research, but it is highly dangerous to trust them too much. It is all too easy to make mistaken inferences unless the process involved is already very well understood." Francis Crick - What Mad Pursuit (1988) "In fact, over the last 100 years, almost all of biology has proceeded independent of evolution, except evolutionary biology itself. Molecular biology, biochemistry, and physiology, have not taken evolution into account at all." Marc Kirschner, Boston Globe, Oct. 23, 2005 "Certainly, my own research with antibiotics during World War II received no guidance from insights provided by Darwinian evolution. Nor did Alexander Fleming's discovery of bacterial inhibition by penicillin. I recently asked more than 70 eminent researchers if they would have done their work differently if they had thought Darwin's theory was wrong. The responses were all the same: No.,,, In the peer-reviewed literature, the word "evolution" often occurs as a sort of coda to academic papers in experimental biology. Is the term integral or superfluous to the substance of these papers? To find out, I substituted for "evolution" some other word – "Buddhism," "Aztec cosmology," or even "creationism." I found that the substitution never touched the paper's core. This did not surprise me. From my conversations with leading researchers it had became clear that modern experimental biology gains its strength from the availability of new instruments and methodologies, not from an immersion in historical biology." Philip S. Skell - (the late) Emeritus Evan Pugh Professor at Pennsylvania State University, and a member of the National Academy of Sciences.
At the 7:00 minute mark of this following video, Dr. Behe gives an example of how positive evidence is falsely attributed to evolution by using the word 'evolution' as a sort of coda in peer-reviewed literature:
Michael Behe - Life Reeks Of Design – video https://www.youtube.com/watch?v=Hdh-YcNYThY Biologists Are Getting to Be Less Reticent About Using the Phrase "Design Principles" - November 28, 2014 Excerpt: The word "design" appears 24 times in the paper. "Selection" appears twice, in the phrase "selective pressure" (one of them is just a repetition from the Abstract). Any form of the word "evolution" appears just once:,,, We see, therefore, that "design" references outnumber evolutionary references eight to one. We also find "machine" or "machinery" four times, "coding" or "encoding" 15 times, "information" (in terms of information to be processed) five times, "accurate" (in terms of sensing accuracy) 11 times, "precision" 29 times, "efficient" four times, and "optimal" or "optimum" 28 times. Taken together, these design words outnumber evolution words 40 to 1. Do the three passing references to evolution/selection add anything to the paper? One would expect to see it in the final Discussion section, but instead, we find these references to design:,,, The paper would lose nothing if its three passing references to evolution/selection were left on the cutting-room floor. All these scientists could do was look at the end product and decide, "Yep, it's fit. It's optimal." http://www.evolutionnews.org/2014/11/biologists_are091531.html
bornagain77_{December 22, 2014
December
12
Dec
22
22
2014
03:07 PM
3
03
07
PM
PDT}

Prev 1 2 3 4

You must be logged in to post a comment.

Leave a Reply