Peer review

Peer review is well and truly bust

Spread the love

Go here for more:

In 2011, after having read several really bad papers in the journal Science, I decided to explore just how slipshod their peer-review process is. I knew that their business depends on publishing “sexy” papers. So I created a manuscript that claimed something extraordinary – that I’d discovered a species of bacteria that uses arsenic in its DNA instead of phosphorus. But I made the science so egregiously bad that no competent peer reviewer would accept it. The approach was deeply flawed – there were poor or absent controls in every figure. I used ludicrously elaborate experiments where simple ones would have done. And I failed to include a simple, obvious experiment that would have definitively shown that arsenic was really in the bacteria’s DNA. I then submitted the paper to Science, punching up the impact the work would have on our understanding of extraterrestrials and the origins of life on Earth in the cover letter. And what do you know? They accepted it!

Remember this when people tell you ID isn’t peer reviewed.

That just means it didn’t pass the local witch test.

38 Replies to “Peer review is well and truly bust

  1. 1
    wd400 says:

    Crazy idea: if the bit you copy from article is not in fact true, then perhaps you should make that clear when you paste it here. Under the “news” heading.

  2. 2
    Sebestyen says:

    Well, the paper wasn’t faked, it was an actual submission. It still contained those faults apparently…

  3. 3
    Jerad says:

    Did news not read down to the part where the author admits this is NOT actually what happened?

    HAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAH

    I’m afraid the News desk is a bit wonky these days.

  4. 4
    sixthbook says:

    From the article:
    “OK – this isn’t exactly what happened. I didn’t actually write the paper. Far more frighteningly, it was a real paper that contained all of the flaws described above that was actually accepted, and ultimately published by Science”.

    Did you guys not read past the part where it says “this is not exactly what happened”?

  5. 5
    Jerad says:

    Did you guys not read past the part where it says “this is not exactly what happened”?

    Yup. No one is saying the peer review system is perfect; it’s administered by human beings!! If you’ve got a better system please present it.

    This example shows up why repeatability is a key issue. One result is interesting but until it’s been verified and duplicated it’s just one result. It could be right, it could be wrong. Hopefully the peer review process catches most of the most egregious mistakes and goofs.

  6. 6
    Barb says:

    Jerad, you may be missing the point. The point isn’t how slipshod the peer review system is, the point is that this paper should never have been published in the first place.

  7. 7
    Jerad says:

    Jerad, you may be missing the point. The point isn’t how slipshod the peer review system is, the point is that this paper should never have been published in the first place.

    Mistakes happen! All the time. How could you possibly designed a system that was run by human beings that avoided all mistakes!! I agree the paper was sub-standard. AND, guess what, it was exposed!

    Until someone comes up with a better system I think we should stick with what we got. Have you got a better system?

  8. 8
    Mung says:

    But Darwin didn’t make a mistake, oh no.

  9. 9
    Jerad says:

    But Darwin didn’t make a mistake, oh no.

    Who told you that? hahahahahahahahahahahahahahahahahahahahahahhaha

    You really need to work on your jokes Mung.

  10. 10
    Jerad says:

    Oh and Barb, I didn’t see where you had suggested a better system than peer review. Maybe your reply got lost in the system. Or maybe you just didn’t bother. Maybe you haven’t actually got a better idea. I don’t know really ’cause you didn’t bother to respond after your attempt to cut down the current system because you think it excludes people you agree with. Have I got that right?

  11. 11
    wd400 says:

    But Darwin didn’t make a mistake, oh no.

    Darwin made plenty of mistakes. Most notably, his epigentic theory of inheritance, the details of his model of speciation and lack pessimistic view of the fossil record’s ability to support his ideas.

    That’s one of many, many reasons to not call evolutionary biology Darwinism.

  12. 12
    wd400 says:

    Jerad, you may be missing the point. The point isn’t how slipshod the peer review system is, the point is that this paper should never have been published in the first place.

    And how should it have been decided that this paper “should never have been published”? Send it out to people working in the field to assess it, perhaps?

  13. 13
    Sebestyen says:

    I don’t think peer review is a bad system in itself (scientists review the work of other scientists) but the execution obviously leaves a lot to be desired. It is unacceptable that a journal accepts a randomly computer generated fake paper, for instance. (Mathgen paper accepted!)

    Personally I think an open peer review system where everything is published (online) and is then open for review from others is better than the “traditional” peer review. Mainly because everything will be published and nothing can be denied prematurely and also because the potential review group could be a lot bigger and much more diverse in comparison to the traditional process.
    Philica may be a step in the right direction but it doesn’t seem to be used very much and I don’t think the reviews should be anonymous.

  14. 14
    Jerad says:

    Personally I think an open peer review system where everything is published (online) and is then open for review from others is better than the “traditional” peer review. Mainly because everything will be published and nothing can be denied prematurely and also because the potential review group could be a lot bigger and much more diverse in comparison to the traditional process.
    Philica may be a step in the right direction but it doesn’t seem to be used very much and I don’t think the reviews should be anonymous.

    Thank you Sebestyen, at least you had the gumption to propose an alternative!!

  15. 15
    wd400 says:

    Good points Sebestyen,

    I would also say, at least within science, peer review is not consider a final arbiter of the worth of a paper. Everyone knows good papers get bumped from glamour journals, and that bad papers are published in many journals. Moreover, the nature of statistics means perfectly executed papers with significant results will often turn out to be “wrong” (i.e. the result won’t reproduce). So, peer review doesn’t end with publication. Once a paper is “out there” scientists will continue to assess it’s worth.

    Since peer review is a guarantee of quality, I don’t think “It’s no peer reviewed” is a good argument against ID. “They haven’t tried to present their results to mainstream scientific audiences” is. If the ID crowd really think they have something to say, they are welcome to write up their results on a per-print server (the arxiv and PeerJ will both accept quantitative biology manuscripts) and let the world know it’s their. Anyone that wants to can then read it, and decide if their is any worth to it (most evolutionary biology journals will accept papers that have appeared as preprints, so it won’t stoo their latter publication either).

  16. 16
    wd400 says:

    I should probably stop being amazed by this, but I still am.

    Does “news” really think it’s OK to have the only quote from a piece to appear here be one the author later explains is not true? In an article about the lack of rigor elsewhere?

  17. 17
    sixthbook says:

    Wd400
    If you change the “I”s with “He”s the quote would be correct. It’s not like a paper of the qualities described in the quote my News wasn’t actually published.

    Your red herrings here are silly.

  18. 18
    sixthbook says:

    Correcting my comment:
    Meant to write “the News” not “my News”

  19. 19
    wd400 says:

    I didn’t journalistic ethics where a “red herring”.

    The quoted passage describes someone going out of their way to fabricate a paper, and deliberately make it a substandard one. That didn’t happen. The quote, which is basically the whole article here, didn’t happen. You can go for the “fake but accurate” defense if you want, but I’m still kind of amazed by the lax standards.

  20. 20
    goodusername says:

    Sixthbook #17:

    If you change the “I”s with “He”s the quote would be correct.

    Well, no, it wouldn’t be correct then either. They didn’t purposely present an asinine study as a test of the peer review process.

    I did take a look at the study in question, but it’s way beyond my ability to judge its quality.

    But looking at what other experts have said, it would appear that the study was, of course, flawed, but the problems were hardly as obvious and ludicrous as made out to be in this article.

    Others tried to duplicate the results and found that while the bacteria do, in fact, need phosphorus, they need less than any other known life, and at the same time they have amazing abilities to survive in high concentrations of arsenic. So they did find something quite unusual, perhaps even unique, and misinterpreted what they were seeing.

    Here’s a more fair take:
    http://www.nature.com/news/stu.....ife-1.9861

  21. 21
    Brent says:

    @ Jerad and wd400,

    I’m pretty appalled that you took the hasty gumption you did to call out News’ “mistake”, but when it turned out that you were in fact mistaken you didn’t say anything. Not an “Oops. Sorry.” Nothing. I glanced that article late at night and thought you were mistaken from my cursory view, but didn’t plan to read in detail and decided not to call you out. Coming back shows that even my cursory skimming was better than your, implied at least, thorough reading. Shameful.

    That is bad form, and if you cannot say so in such a small matter, who should trust you to be honest about a weighter matter that, uh, actually matters?

    As for peer review, it is a nightmare. Good idea, but it’s broken. Why? Because people, even the lab-coat people, are selfish, looking-out-for-number-one, egotistical . . . well, in short, they’re sinners just like everyone else.

    I’m currently helping a Japanese scientist to revise a paper to be accepted to a journal. It is astonishing the things she has to consider and worry about in the peer review process in order to have a snowball’s chance; childish people protecting their own precious research even though it wasn’t very good when initially published, and has been proven wrong by her new work. These guys care about their name in lights more than about truth.

    And further, how can anyone seriously defend the peer review process by saying, “well, mistakes happen”? I mean, it is true that mistakes happen, but when the “mistake” literally undermines the whole reason for the existence of the enterprise undertaken, something more serious is the issue. The process is specifically for what it failed spectacularly to do. There is ZERO excuse.

  22. 22
    wd400 says:

    I’m sorry, what? The thing in the quote didn’t happen, doesn’t that seem important?

  23. 23
    Mark Frank says:

    The argument of the article is that peer review lets things through which it shouldn’t. I have not seen anything about how peer review excludes things which is should let in i.e. peer review is too lax. So if the message of the article is right and ID can’t even get through peer review it is in a very bad state.

  24. 24
    Brent says:

    OK – this isn’t exactly what happened. I didn’t actually write the paper. Far more frighteningly, it was a real paper that contained all of the flaws described above that was actually accepted, and ultimately published, by Science.

    So, we’re supposed to let you off on a technicality; the article author isn’t the one who wrote the paper, but someone else actually did and it was actually published in Science?

    And what does that change about anything we were supposed to take away from this post by News? Oh! Nothing. Right.

  25. 25
    wd400 says:

    but someone else actually did and it was actually published in Science?

    No.

  26. 26
    Brent says:

    Hear no evil, see no evil, eh Mark?

    No, the “mistakes” only happen one way. When stuff that is supposed to be weeded out is let through, it ironically supposes to confirm the materialistic paradigm, and just slipped through the cracks. When stuff isn’t let through, it is because it is “rightly” weeded out, and it’s just a coincidence that it refutes the materialistic paradigm. No design in that, either, huh?

  27. 27
    Brent says:

    wd400,it looks like you’ll have to take this up with the author of the article quoted. He says “yes”, you say “no”.

  28. 28
    Mark Frank says:

    #26 Brent

    I don’t follow your argument. The point is that the peer review process sets the bar too low. So poor stuff gets published. However, there is no reason to suppose good stuff gets rejected. So far this is independent of whether the stuff is materialist or not.

    Now we note that almost no ID papers get published while plenty of “materialist” stuff does get published (including some rubbish). So it seems like the ID community is unable to produce much material that even gets over this low bar.

    If stuff that is supposed to be weeded gets through it doesn’t confirm anything. If stuff is rejected then it must be pretty poor that’s all.

  29. 29
    wd400 says:

    Eisen does not claim that the arsenic life paper was fabricated to serve as a test of the glamour mags’ peer review process. It’s really very simple. The action described in the quoted section never happened. I dont’ think I’m crazy in thinking that this is pretty “slipshod” reporting on behalf of News. I am staggered anyone would defend it.

  30. 30
    Brent says:

    Mark, the point is the “low bar” of which you speak is set low for the “right people with the right papers”, just as it might be set low in the Special Olympics (though of course acceptable in that case). So, people can do crappy work and, as long as the content pushes the right metaphysical buttons along the way, it is very likely to get accepted. But, high quality work that pushes the wrong metaphysical buttons does not get accepted. I am not personally willing to search and post the examples, though immediately Granville Sewell’s work comes to mind as a case of the latter.

  31. 31
    Mark Frank says:

    It seems to me that most of “News” OPs fall into one of two categories – stuff which didn’t happen and stuff that isn’t new.

  32. 32
    Brent says:

    wd400, I was serious about you and Jerad using bad form. The point I wish to make now is that I am quite willing to lead the way if I’m missing something here.

    The article uses the arsenic paper to segue into the main topic for discussion. Either the arsenic paper was written and published by Science (or one sufficiently like it so as to garner the word “it” being used to say it really was done by someone), or not. So, yes, it really is very simple.

    If it was written and published, then you are wrong. If it was not written, the article itself is either a lie, misleading, and/or poorly written.

  33. 33
    wd400 says:

    There was a real arsenic life paper, which has been gone over a million times, including here. It had flaws, I don’t think most would say it was as bad as Eisen is saying.

    It was not fabricated as “trap” paper to test peer review

    Eisen doesn’t claim that it was, calling it a “real paper”.

    Again, the thing in the quote didn’t happen.

  34. 34
    Brent says:

    Sorry! Typhoon is knocking and computer is going down now. Water coming through the window!

  35. 35
    Alan Fox says:

    Thepaper for those unable to put éarsenic paper” in Google

  36. 36
    Brent says:

    wd400, I now see what you mean. I took “real paper” only as a means of Eisen to say that, though he didn’t do the deed, someone else had; not that it was a serious paper not being used as a gotcha. I thought he only meant that someone else had written this gotcha paper.

    Now if News already knew this, I don’t know why this post was written the way it was. If News didn’t know, I think the mistake is a pretty easy one to make.

    So, I take back my “bad form” and “shameful” comments concerning the “fake/real paper” issue.

    The interesting thing is, it seems Eisen is saying it doesn’t really matter at the end of the day if the paper was “real” or “fake”. It could have been faked, and according to him was bad enough to have been faked, and that Science is being pretty cheeky with their high-ground stance since a paper as bad, but “real”, was not so long ago graced in their own pages: and that while the “sting” paper study was without control and therefore would possibly have been accepted by brand name journals as well. So, not only do they have shame in their recent history, they have open shame in publishing this very story and keeping their own sins hush-hush.

  37. 37
    Barb says:

    Jerad writes,

    h and Barb, I didn’t see where you had suggested a better system than peer review. Maybe your reply got lost in the system. Or maybe you just didn’t bother. Maybe you haven’t actually got a better idea. I don’t know really ’cause you didn’t bother to respond after your attempt to cut down the current system because you think it excludes people you agree with. Have I got that right?

    Settle down, Jerad. Getting sarcastic and rude doesn’t help your case any. And you haven’t really gotten anything right.

    wd400 @ 12:

    And how should it have been decided that this paper “should never have been published”? Send it out to people working in the field to assess it, perhaps?

    No, this would have been decided by the peer review committee being intelligent enough and knowledgeable enough on the subject at hand to realize that it’s junk science and not publish it. Duh.

  38. 38
    Phinehas says:

    From later in the article (and, in my opinion, supporting the point News was trying to make, though I think the suggested clarification would have helped avoid confusion):

    And the real problem isn’t that some fly-by-night publishers hoping to make a quick buck aren’t even doing peer review (although that is a problem). While some fringe OA publishers are playing a short con, subscription publishers are seasoned grifters playing a long con. They fleece the research community of billions of dollars every year by convincing them of something manifestly false – that their journals and their “peer review” process are an essential part of science, and that we need them to filter out the good science – and the good scientists – from the bad. Like all good grifters playing the long con, they get us to believe they are doing something good for us – something we need. While they pocket our billions, with elegant sleight of hand, then get us to ignore the fact that crappy papers routinely get into high-profile journals simply because they deal with sexy topics.

    Since it is common for anti IDists to haul out the “peer review” canard, it seems to me that this article has something very pertinent to say despite the attempts to distract from it. According to Eisner (no friend of ID), the peer review process is not an essential part of science, nor should it be trusted as way to filter out good science from bad. Will anti IDists around these parts agree with Eisner, or will they ignore the evidence he provides in support of this view? I suspect they are too enamored with their “peer review” objection to be anything but dismissive toward the pertinent issues raised.

Leave a Reply