Peer review

Peer review, mere review, and smear review

Andrew Sibley here discusses a thoughtful article by Fred Pearce in the Guardian (02 February 2010) on the climate change scandal, an article which had also been mentioned to me by a kind reader recently. The article takes a critical look at peer review, a well-justified critical look in my view.

I have written about the problem with peer review here, and would recommend Frank Tipler’s paper on the subject.

The basic problem is that the peer review process, intended to enforce quality, can end up enforcing mere orthodoxy or, worse, mediocrity. Or worst of all, as in the now-famous climategate e-mails, it can lead to a classic “bunker” mentality.

I would be inclined to treat all science-based dissent as legitimate. The mere fact that some scientists cannot replicate others’ work or support their conclusions is not evidence of incompetence or dishonesty. It may lead to useful corrections or valuable new information.

Of course, if someone claims that climate change is caused by space aliens, an evil plot by a minority group, or proof that Jesus is coming again soon, I would say, please, this is not science. Science is about evidence from nature.

I was trying to remember recently what peer review reminded me of, and then I suddenly remembered: Read More ›

Here the controversy erupted over an article critiquing estimates of  war deaths.

Researchers from Canada, the UK and Sweden have slammed the influential British Medical Journal (BMJ) for publishing an error-filled study on global war deaths, refusing an equivalent rebuttal article and having a flawed peer-review process.

Apparently, the contested article took issue with the fact that Oslo’s International Peace Research Institute data show that global war deaths “declined by more than 90 per cent between 1946 and 2002.”

“This is not some trivial academic disagreement,” says Andrew Mack, director of the Simon Fraser University-based Human Security Report Project (HSRP), which published a detailed critique of the BMJ’s claims in the December issue of the Journal of Conflict Resolution (JCR).

“Accurate statistics on the health impacts of war are critically important not just for researchers but also for humanitarian organizations whose assistance programs save millions of lives around the world.”

The BMJ doesn’t deny the problem:

“But the BMJ is well aware that its peer review process is flawed,” says Spagat. “A recent study, whose authors include the journal’s current editor, revealed that, on average, only a third of the ‘major errors’ deliberately inserted in a BMJ article were picked up by reviewers.”

In what other line of work would such incompetence be accepted? Would you like your electrician to achieve only this level of competence? He only “gets” one third of the electrical safety hazards in your home?

And remember, if you live in the UK, your taxes pay for these scholars to “do their thing.”

Adds Mack: “There appears to be no way of effectively rebutting BMJ articles that contain unwarranted — and damaging — critiques of the work of other scholars.

A couple of years back, I wrote on the problem of peer review: Often, it is simply the way establishment hacks prevent competition from new information and new interpretations.

Re war deaths, two notes: Read More ›