Peer review

The replication crisis in science grinds on into another decade

Spread the love

Maybe the system is unreformable:

A recent write-up by Alvaro de Menard, a participant in the Defense Advanced Research Project’s Agency’s (DARPA) replication markets project (more on this below), makes the case for a more depressing view: The processes that lead to unreliable research findings are routine, well understood, predictable, and in principle pretty easy to avoid. And yet, he argues, we’re still not improving the quality and rigor of social science research.

While other researchers I spoke with pushed back on parts of Menard’s pessimistic take, they do agree on something: a decade of talking about the replication crisis hasn’t translated into a scientific process that’s much less vulnerable to it. Bad science is still frequently published, including in top journals — and that needs to change.

Most papers fail to replicate for totally predictable reasons.

Kelsey Piper, “Science has been in a “replication crisis” for a decade. Have we learned anything?” at Vox

Systems can be unreformable when there is no compelling reason to pursue reform. As when, for example, bad stuff is funded right along with good stuff.

A view worth looking at is Robert J. Marks’s Why it’s so hard to reform peer review. Reformers are battling numerical laws that govern how incentives work. Know your enemy! Goodhart’s Law, for example, captures the unintended effect of using numerical metrics as goals: “When a measure becomes a target, it ceases to be a good measure.”

8 Replies to “The replication crisis in science grinds on into another decade

  1. 1
    martin_r says:

    similar article from 2017 (BBC):

    “Most scientists ‘can’t replicate studies by their peers’
    “Science is facing a “reproducibility crisis” where more than two-thirds of researchers have tried and failed to reproduce another scientist’s experiments, research suggests. ”

    https://www.bbc.com/news/science-environment-39054778

    or this one:

    “107 cancer papers retracted due to peer review fraud

    The journal Tumor Biology is retracting 107 research papers after discovering that the authors faked the peer review process. This isn’t the journal’s first rodeo. Late last year, 58 papers were retracted from seven different journals— 25 came from Tumor Biology for the same reason.”

    https://arstechnica.com/science/2017/04/107-cancer-papers-retracted-due-to-peer-review-fraud/

    and so on…

    but don’t forget… trust science!!!!! trust scientists !!!!! YOU MUST !!!

  2. 2
    polistra says:

    Replication, like plagiarism and censorship, is a fake concern.

    The real problem is that about 2/3 of papers weren’t worth publishing ONCE, let alone TWICE. Tenure encourages a vast number of small variations on old themes, which aren’t really interesting and aren’t intended to break new ground. Re-trying these small variations simply doubles the waste of effort and money and brainpower.

  3. 3
    Seversky says:

    First, it was science that exposed its own replication crisis.

    Second, if you’re smart, you can learn more from getting things wrong than from getting things right.

    Third, to everyone gloating over the replication crisis, of you have a better idea then let’s hear it.

  4. 4
    AaronS1978 says:

    I would like to correct you science did not expose anything, it was human beings, some practicing science, some not, with a rudimentary amount of honesty in them, that exposed it

    And they weren’t getting things wrong, many were just factually being dishonest (again oxytocin a great place to start for examples of this)

    And nobody’s really gloating about the replication crisis, it’s the fact, and I stated this on another post, that you have these very ugly people utilizing science and claiming to be experts and telling us how to live our lives you have to trust then, and then there is is lovely crisis that calls their claim to question (this is not science as a whole, but the media hungry ones that often tell you how you need to live)

    If there is a better answer it’s called reworking the system to were money doesn’t have the leverage it does, politics doesn’t have a say,
    And we have to screen for honest people

    Shit, those very same things would also work for our current political system

    But unfortunately that requires us getting the people that are in power out and replacing them with a new generation of individuals that have not been corrupted to the point of no return

  5. 5
    Latemarch says:

    AaronS1978@4

    If there is a better answer it’s called reworking the system to were money doesn’t have the leverage it does, politics doesn’t have a say,
    And we have to screen for honest people

    LOL!
    Your really bucking human nature there. Kind’a like trying to swim up a waterfall.

  6. 6
    AaronS1978 says:

    @ latemarch

    WHAT!? What ever do you mean!?!?!?!?!?!??
    ARE YOU IMPLYING HUMANS CANT BE HONEST?!?!!?!!??!!??
    I’m shocked and dismayed at your pessimism

    Sorry you might need to wipe all that sarcasm off I might’ve used a bit much

  7. 7
    EDTA says:

    From the article linked, there are more things that should be done:

    We also need to change the fact that those better papers aren’t cited more often than bad papers; that bad papers are almost never retracted even when their errors are visible to lay readers; and that there are no consequences for bad research.

    And, I happily cite this quote:

    Additional research has established that you don’t even need to poll experts in a field to guess which of its studies will hold up to scrutiny. A study published in August had participants read psychology papers and predict whether they would replicate. “Laypeople without a professional background in the social sciences are able to predict the replicability of social-science studies with above-chance accuracy,” the study concluded, “on the basis of nothing more than simple verbal study descriptions.”

    Meaning that although our comments here are not the final word on whether social science findings are valid or not, it wouldn’t hurt to take them into account.

  8. 8
    ET says:

    seversky:

    First, it was science that exposed its own replication crisis.

    Science didn’t do anything. Scientists and other people do.

    Second, if you’re smart, you can learn more from getting things wrong than from getting things right.

    It hasn’t helped you nor any other evo. And all you are is wrong.

    Third, to everyone gloating over the replication crisis, of you have a better idea then let’s hear it.

    Stop trying to promote bad science, uh.

Leave a Reply