Uncommon Descent Serving The Intelligent Design Community

The buzz now is all for replication papers but what happened when researchers submitted one to Nature?

Denyse O'Leary
May 11, 2018
Intelligent Design, Peer review, Science
Share
Facebook
Twitter
LinkedIn
Flipboard
Print
arroba Email

From Mante Nieuwland at Retraction Watch:

On April 10th 2018, eLife published the first large-scale direct replication study in the field of cognitive neuroscience, co-authored by 22 colleagues and myself. This publication detailed a replication effort that spanned 9 laboratories and attempted to replicate a high-impact 2005 publication in the prestigious journal Nature Neuroscience from DeLong, Urbach and Kutas (from hereon referred to as DUK05). People often ask why our replication study was not published in Nature Neuroscience, especially in light of its recent public commitments to replication research (here and here). It certainly wasn’t for our lack of trying.

In this post, I offer a behind-the-scenes account of what happened when we tried to replicate DUK05 and submitted our results to Nature Neuroscience, illustrating some of the obstacles faced by researchers who perform and try to publish replication research. These experiences are not unique by any means (for examples, see here and here). I am sharing our experiences openly in the hope that Nature and other publishers can become more transparent about their policies regarding replication research, and I hope that our experiences will serve and support anyone with an interest in replication research.More.

Treat yourself to an extra virtual cappuccino if you guessed that parts of the boffo seminal study could not be replicated. And then…

Sounds like everyone wants replication in principle but no one wants to subject their own papers to the risk. After all, a paper that no one has even tried to replicate exists in a timeless zone of “never disconfirmed” and can go on being buffo and a basis for policy, without being factually correct. Why mess with that?

Keep up to date with Retraction Watch.

See also: Missing data hinder replication in AI studies too?

and

Scientists should not accept unreplicated results (Yawn.)

Comments
Maybe Amazon could start some journals that a researcher could self publish to. Kind of like Kindle but for researchers. Require all data. All software if you write it yourself. Reviews would have a way of being responded to by the researcher. It's worked well for indie authors. Latemarch
Seversky,
Perhaps some sort of online repository which would allow researchers to discover any studies which replicate their findings – or not – as the case may be?
Perhaps. But the problem runs deeper than the publication of repeated research. I have a friend who was unable to publish any of the work from his masters degree (except as a thesis) because the outcome of the work could not conclude anything. This alone is valuable information, but no way to communicate this. Allan Keith
Perhaps some sort of online repository which would allow researchers to discover any studies which replicate their findings - or not - as the case may be? Seversky
Repeatability is obviously a major problem in science. Not that repeatability isn't important but that it is often difficult to get it published. A repeat that obtains identical results will often get a response that the paper shows nothing new. A repeat that is inconclusive will seldom be submitted for publication, as is the case with original research that is inconclusive. Sadly, these inconclusive results are extremely valuable in that they may rule out certain avenues of research. By not publishing them, other researches may waste time and effort (and money) following the same line of research. Sadly, I don't know what the answer is. I doubt that journals dedicated to repeated research will find an audience. Allan Keith

Leave a Reply