Intelligent Design

Science “sting” shows peer review catastrophically failing

Spread the love

Remember the Sokal Hoax? A physics professor manages to sneak in a completely garbage paper to a “postmodern cultural studies” journal? Well, if you thought that science journals were immune to this sort of thing – or even more often than not reliable – then get ready to have some of your faith in the modern academia broken up a bit.

Who’s Afraid of Peer Review? documents the story, and here’s a bit of the opening:

On 4 July, good news arrived in the inbox of Ocorrafoo Cobange, a biologist at the Wassee Institute of Medicine in Asmara. It was the official letter of acceptance for a paper he had submitted 2 months earlier to the Journal of Natural Pharmaceuticals, describing the anticancer properties of a chemical that Cobange had extracted from a lichen.

In fact, it should have been promptly rejected. Any reviewer with more than a high-school knowledge of chemistry and the ability to understand a basic data plot should have spotted the paper’s short-comings immediately. Its experiments are so hopelessly flawed that the results are meaningless.

Sounds like some pretty obvious garbage. But hey, just how many journals could have possibly let this one slip by the gates?

By the time Science went to press, 157 of the journals had accepted the paper and 98 had rejected it. Of the remaining 49 journals, 29 seem to be derelict: websites abandoned by their creators. Editors from the other 20 had e-mailed the fictitious corresponding authors stating that the paper was still under review; those, too, are excluded from this analysis. Acceptance took 40 days on average, compared to 24 days to elicit a rejection.

Now, it was open access journals that were targeted here. But the results aren’t all that encouraging for ‘traditional’ journals either. With these results in mind, here’s a question worth considering. If an obviously nonsense paper really can make it into so many journals, what are the odds that papers with more subtle but still serious flaws are slipping past the gates?

18 Replies to “Science “sting” shows peer review catastrophically failing

  1. 1
    lifepsy says:

    Just tally up the number of papers that base conclusions on common descent/evolutionary assumptions, and you have the number of papers and corresponding journals that have allowed philosophy/religion studies to be mixed with empirical methodology… must be in the tens of thousands at least.

  2. 2
    NickMatzke_UD says:

    Well, one thing this helps explain is a lot of the recent “peer-reviewed” ID articles which are typically published in The Journal of I’ve Never Heard of It, Volume Less Than 10.

  3. 3
    Jerad says:

    Well, one thing this helps explain is a lot of the recent “peer-reviewed” ID articles which are typically published in The Journal of I’ve Never Heard of It, Volume Less Than 10.

    Well jeeze Dr Matzke, aren’t peers people who have a similar educational background and who hold similar views?

    There should be peer (in the way I’ve described it) reviewed journals for Astrology and UFO-ology and Reiki and Homeopathy and Rolfing and Acupuncture and Baraminology and Flat Earthology (or whatever it’s called) and Erich Von Daniken and Diving and Reflexology and ID. How else are they going to get a fair review except by others who know what they’re talking about? ‘Cause clearly you materialists are completely shut off to any ideas that threaten your state-sponsored hegemony. With your Ockham’s Razor and your parsimonious explanations that don’t involve special pleading or assuming causes and forces which haven’t been proven to exist. You guys just lack any kind of faith or imagination.

  4. 4
    NickMatzke_UD says:

    Homeopathy? UFOs? I’m glad you’re on their side!

  5. 5
    TheisticEvolutionist says:

    What peer-reviewed publications do you have out Matzke?

  6. 6
    franklin says:

    TE: What peer-reviewed publications do you have out Matzke?

    google scholar is your friend!

  7. 7
    Ian Thompson says:

    f: google scholar is your friend!

    Do you know how to filter out the newspaper-like ‘reviews’, to just look at the actual research?

  8. 8
    nullasalus says:

    So, an incident of widespread failure of peer review is found, and the first thing Nick Matzke does is stammer, “B-b-b-but… Intelligent Design! *whimper*”.

    Behold, ladies and gentlemen – the modern defender of science. Highlight a big problem with the field of science, and he doesn’t want to talk about it. He doesn’t find it discouraging. Heck, he doesn’t even acknowledge it as a problem. The first, and only, thing that matters to him is whether or not there’s a way he can knock his political and religious enemies.

    And you know what? That sort of mentality is probably one reason why peer review has been failing. Journal editors glance at the paper, check to see if design or intelligent agents are mentioned. If so? More often than not, rejection pile! If not? Meh. Does anything look out of the ordinary? No? Approved. So long as it’s not advancing intelligent design or questioning global warming, it doesn’t really matter.

  9. 9
    franklin says:

    nullasalus: If so? More often than not, rejection pile! If not? Meh. Does anything look out of the ordinary? No? Approved. So long as it’s not advancing intelligent design or questioning global warming, it doesn’t really matter.

    From your comments I have to wonder if you even read the article the OP cites.

    There is certainly a problem with the proliferation of shoddy open-access journals where the only publication requirement is having the check clear.

    All science journals are not created equal and anyone can easily see the ‘scam’ nature of many of these open-access journal. Indeed, the article cites a very clear and unambiguous example with its’ comparison of The American Journal of Polymer Science with Journal of Polymer Science.

    Open-access, on paper, is a good idea but it’s not surprising that these lowest-tiered of journals are gaming the system for profit with no regard for content.

  10. 10
    nullasalus says:

    From your comments I have to wonder if you even read the article the OP cites.

    Of course I did. For instance, there’s nothing in there about Intelligent Design. Matzke, however, may not have read it – or read it and didn’t care since it didn’t knock his political opponents.

    I’m sure he’ll make a great peer reviewer.

    There is certainly a problem with the proliferation of shoddy open-access journals where the only publication requirement is having the check clear.

    Funny. The article indicates that this problem is a bit more far reaching. Here’s another snippet:

    One might have expected credible peer review at the Journal of Natural Pharmaceuticals. It describes itself as “a peer reviewed journal aiming to communicate high quality research articles, short communications, and reviews in the field of natural products with desired pharmacological activities.” The editors and advisory board members are pharmaceutical science professors at universities around the world.

    …And yet this journal fell for the hoax.

    Easy to see why:

    But even when editors and bank accounts are in the developing world, the company that ultimately reaps the profits may be based in the United States or Europe. In some cases, academic publishing powerhouses sit at the top of the chain.

    But I agree with your basic sentiment: peer review is no guarantee that a paper is even basically credible. And there’s no lack of unethical scientists or academics who are more than willing to put science aside in the name of profit or personal/political motivations. Some healthy skepticism of scientists and scientific claims is entirely reasonable.

  11. 11
    franklin says:

    nullasalus: …And yet this journal fell for the hoax.

    which elicited this response from the editor after finding out that the managing editor left in charge and unsupervised was not doing his job.

    But the editorial team of the Journal of Natural Pharmaceuticals, headed by Editor-in-Chief Ilkay Orhan, a professor of pharmacy at Eastern Mediterranean University in Gazimagosa, Cyprus, asked the fictional Cobange for only superficial changes to the paper—different reference formats and a longer abstract—before accepting it 51 days later. The paper’s scientific content was never mentioned. In an e-mail to Science, managing editor Mueen Ahmed, a professor of pharmacy at King Faisal University in Al-Hasa, Saudi Arabia, states that he will permanently shut down the journal by the end of the year. “I am really sorry for this,” he says. Orhan says that for the past 2 years, he had left the journal’s operation entirely to staff led by Ahmed. (Ahmed confirms this.) “I should’ve been more careful,” Orhan says.

    it also seems to me that Matzke’s comment was directed at the problem of having a large editorial staff of a journal which publishes not much else than the contributions of the editorial staff. That type of incestuous relationship between manuscript authors and the publishing staff raises many questions on the validity of anything that is published. In fact scientists who have ‘peer-reviewed’ these articles after they were published have found many errors which should have been caught bringing into question the quality of the peer-review given the manuscripts.

  12. 12
    nullasalus says:

    which elicited this response from the editor after finding out that the managing editor left in charge and unsupervised was not doing his job.

    Woah woah woah. You mean when a journal is caught red-handed accepting a hoax paper, they promise to take steps to deal with it?

    Well, that’s inspiring. Problem solved from their end. Makes me wonder why anyone would even bother publishing the article I linked. Bunch of dirty, lousy troublemakers, harping about a problem everyone knows about and is already solved.

    it also seems to me that Matzke’s comment was directed at the problem of

    Nick Matzke’s comment turned a complete blind eye to the actual problem – widespread problem, in fact – with peer review, corrupt scientists and academics, and more. Because that topic makes him feel uncomfortable. The whole point of ‘defending science’ for Matzke is ‘attacking ideas he dislikes or even fears’. Peer review failure and the like? Booooring. Worse, threatening. People may end up having some healthy skepticism of scientists, and if they do that, that makes his culture war job more difficult.

    Why in the world are you trying to run defense for him? The man spoke for himself and showed his hand – he doesn’t really care about actual problems in the scientific process. He cares about bashing his political enemies. Trying to parse his words as if you were engaged in interpretation of a holy work is silly.

  13. 13
    franklin says:

    nullasulas: Nick Matzke’s comment turned a complete blind eye to the actual problem – widespread problem, in fact – with peer review, corrupt scientists and academics, and more. Because that topic makes him feel uncomfortable.

    I have no doubt that Nick is as aware as I am of what constitutes a quality, and reliable, journal for submission of manuscripts. It is what drives to submit our manuscripts to top-tier journals rather than ‘publication mills’.

    The incestuous nature of the editorial staff at the latest ID ‘journal’ is every bit as serious as that presented by the proliferation of open-access journals and I am surprised that you are giving them a pass. The question raised is one of the quality of peer review in such an environment. do you really think that the editorial staff at BioComplexity would ever reject a manuscript from one of the editors let alone anyone else who might happen to submit for publication? The journal, certainly, has little credibility as far as peer-reviewed journals are concerned.

    FYI, nullasalus I don’t need to defend NIck at all in fact I just pointed out a serious concern with the leading ID journal that pretty much everyone acknowledges….outside of ID proponents that is. You seemed confused as to why he would make such a comment and I was hoping to clarify it as the context of the comment was apparent, well at least to me maybe not you.

  14. 14
    nullasalus says:

    I have no doubt that Nick is as aware as I am of what constitutes a quality, and reliable, journal for submission of manuscripts.

    ‘One that doesn’t accept any papers Nick dislikes or thinks would prompt anyone to ever question ideas he wants people to accept on faith.’

    The incestuous nature of the editorial staff at the latest ID ‘journal’ is

    …utterly irrelevant to the subject, as desperate as both you and Nick are to change it, Franklin.

    The article dealt with the poor state of peer review, my friend. This is about the state of science in general, particularly in regards to ‘peer review’. No ID was mentioned in the article, and it’s not even about bias in favor of particular articles or worldviews, but apathy about the scientific content of papers.

    As with Matzke, you only care about science insofar as you can link it to your political and social enemies. If they’re not involved? Who cares.

    Again – behold the mighty defenders of science.

    FYI, nullasalus I don’t need to defend NIck at all

    And yet here you are, defending him and trying to talk about the deep and thoughtful commentary he clearly meant with what amounted to a one line ‘I HATE ID!!’ comment.

    You seemed confused as to why he would make such a comment

    I wasn’t confused. I thought it beautifully illustrated his dislike and lack of interest in science when it doesn’t suit his needs. What a surprise – the same applies to you.

    Abuses of science and peer review only matter to you insofar as you can link it to thoughts you dislike. Otherwise, who cares? It doesn’t impact your religious beliefs, so the interest is gone.

  15. 15
    franklin says:

    nullasalus: The article dealt with the poor state of peer review, my friend.

    Which is why the situation at BioComplexity is an issue for consideration and certainly spot on as far as the OP is concerned. Unless, that is, you don’t consider BioComplexity to be a science journal.

    This is about the state of science in general, particularly in regards to ‘peer review’.

    then why are you shirking the issue that BioComplexity brings to the table, i.e., peer-review?

    Abuses of science and peer review only matter to you insofar as you can link it to thoughts you dislike.

    I can’t see that you would be able to type that with a straight face given your ignoring the very issue you are carping about except now you would be faced with commenting on the issue of peer-review as it pertains to Bio Complexity.

    What is the life-span of ID journals?

    Otherwise, who cares?

    Evidently not you my culture warrior friend!

  16. 16
    nullasalus says:

    Which is why the situation at BioComplexity is

    Irrelevant. Hence why there is no, none, zero, nyet mention of this in the original article.

    But by all means, I know you have far more important things to worry about than science, or widespread failure of peer review. That’s only interesting if you kinda-sorta can possibly draw a line to your hated political and religious enemies.

    then why are you shirking the issue that

    Because it’s irrelevant? The article didn’t single out ‘intelligent design proponents’. Nor did they complain about peer reviewers’ supposed ideological agendas. This was not a case of peer reviewers letting in papers they agreed with – it was their not even bothering to check out fairly bland and inoffensive papers, period.

    But really, keep avoiding that topic as long as you like. I’ll keep drawing attention back to it and failing to take your bait – and every time I do, you’re illustrating how little you care for science.

    I can’t see that you would be able to type that with a straight face given your ignoring the very issue you are carping about

    Ignoring it? Little man – are you aware that I’m the resident ID skeptic? I flat out say I don’t think ID is science whenever the question comes up in a relevant thread. While I think ID thoughts and insights are valuable, even compelling, I don’t consider them (or their mirror images, the design deniers) to be scientific.

    But I do think that there are some serious problems in science, and serious reason to be reasonably skeptical of our much-proclaimed modern scientific processes. So when there’s a catastrophic failure of peer review, it’s worth talking about.

    You, meanwhile, don’t care unless you get to talk about your hated enemies. That’s where you and I differ, Franklin. I actually am concerned with science and reason in general. For you, it’s only interesting when it can be a means to a political end.

    Evidently not you

    Funny. I’m the only one between the two of us trying to keep the topic on the actual article’s point – the widespread failure of peer review when it comes to scientific papers. So I certainly show I care about that.

    You want to talk about an unrelated topic. But don’t worry – I’m sure Nick Matzke is glad to have you lapping at his boots. Why, maybe someday he’ll pat you on the head in public and you’ll get butterflies in your tummy – golly, complimented by a blogger who’s almost 1/10th as popular as PZ Myers!

    Me? I don’t care for getting patted on the head.

  17. 17
    Mung says:

    I don’t consider them (or their mirror images, the design deniers) to be scientific.

    As a fully certified ID proponent and design denier I am the only one actually “doing science.”

  18. 18
    Paul Giem says:

    I have a suggestion for franklin and Nick Matzke. Why don’t you do your own little sting operation? Have someone submit an article that favors ID, but has demonstrable scientific errors of a kind that both ID proponents and opponents would agree are errors. Then, as a control, submit an article that criticizes ID, but has the same errors of widely agreed fact. Then submit the same articles to other journals. Then finally publish all the results. I’d be interested to see the results.

    It might be helpful to have an ID proponent look over the articles, just so you know that the errors are in fact widely agreed upon. In fact, it would strengthen your conclusion if you have previous ID reviewers agree that some proposition is in error, and the journal takes it (assuming the journal does indeed take it). Perhaps, if you are afraid they will tip off the journal, you can avoid telling them exactly why you are having them review the article.

Leave a Reply