For readers following the peer review controversies (see, for example, “Another Nobelist denounces peer review”), here’s a 2006 article for your files, from the Journal of the Royal Society of Medicine:
Peer review: a flawed process at the heart of science and journals
Richard Smith,
Chief Executive, UnitedHealth Europe
Peer review might also be useful for detecting errors or fraud. At the BMJ we did several studies where we inserted major errors into papers that we then sent to many reviewers.3,4 Nobody ever spotted all of the errors. Some reviewers did not spot any, and most reviewers spotted only about a quarter. Peer review sometimes picks up fraud by chance, but generally it is not a reliable method for detecting fraud because it works on trust. A major question, which I will return to, is whether peer review and journals should cease to work on trust.
As Einstein might say … hmmm
People have a great many fantasies about peer review, and one of the most powerful is that it is a highly objective, reliable, and consistent process. I regularly received letters from authors who were upset that the BMJ rejected their paper and then published what they thought to be a much inferior paper on the same subject. Always they saw something underhand. They found it hard to accept that peer review is a subjective and, therefore, inconsistent process. But it is probably unreasonable to expect it to be objective and consistent. If I ask people to rank painters like Titian, Tintoretto, Bellini, Carpaccio, and Veronese, I would never expect them to come up with the same order. A scientific study submitted to a medical journal may not be as complex a work as a Tintoretto altarpiece, but it is complex. Inevitably people will take different views on its strengths, weaknesses, and importance.
One might reasonably object that all these painters are “for the ages,” in the view of art historians, but we would surely encounter (a more frenetic version of) the same effect at a local ice sculpture show and contest.
So, the evidence is that if reviewers are asked to give an opinion on whether or not a paper should be published they agree only slightly more than they would be expected to agree by chance. (I am conscious that this evidence conflicts with the study of Stephen Lock showing that he alone and the whole BMJ peer review process tended to reach the same decision on which papers should be published. The explanation may be that being the editor who had designed the BMJ process and appointed the editors and reviewers it was not surprising that they were fashioned in his image and made similar decisions.)
So let’s say a paper is submitted to a biology journal that 1) addresses evolution but 2) is not just another Darwinography.
What are the chances it faces not so much a review by peers as a hanging jury of thugs protecting their turf?
Question: Many people have heard Michael Shermer’s self-satisfied pronouncements about the peer review process in 2011:
Belief-dependent realism is driven even deeper by a meta-bias called the bias blind spot, or the tendency to recognize the power of cognitive biases in other people but to be blind to their influence on our own beliefs. Even scientists are not immune, subject to experimenter-expectation bias, or the tendency for observers to notice, select and publish data that agree with their expectations for the outcome of an experiment and to ignore, discard or disbelieve data that do not.
This dependency on belief and its host of psychological biases is why, in science, we have built-in self-correcting machinery. Strict double-blind controls are required, in which neither the subjects nor the experimenters know the conditions during data collection. Collaboration with colleagues is vital. Results are vetted at conferences and in peer-reviewed journals. Research is replicated in other laboratories. Disconfirming evidence and contradictory interpretations of data are included in the analysis. If you don’t seek data and arguments against your theory, someone else will, usually with great glee and in a public forum. This is why skepticism is a sine qua non of science, the only escape we have from the belief-dependent realism trap created by our believing brains.
Would he really say the same things today?
But come to think of it. Where else can a loyal Darwinist go?
Follow UD News at Twitter!