Ethics Peer review

Surprise: Science thrives when people can admit they didn’t prove something

Spread the love
What’s hot? What’s not?/Niklas Bildhauer, Wikimedia

For decades, researchers have complained that one of the problems that compromises research today is null findings – that is, if you don’t prove your point, you’re a failure, so you bury it (the file drawer problem). But that attitude shows poor collective reasoning skills. Showing which answers are wrong is, logically, a step on the road to right answers.

A proposed reform has been registering studies and peer reviewing the protocols, so that whatever the researchers find will count as a finding. They don’t have to find something when there’s nothing, as long as the hunt is agreed to be worthwhile in principle, justifiable. What happened when it was tried?

To see whether registered reports increase the frequency at which null results are reported, psychologists Chris Allen and David Mehler analysed the outcomes of 113 registered reports in the biomedical and psychological sciences.

The pair identified 296 discrete hypotheses across those studies, and found that, overall, 61% of these were not supported by the results that those studies later published. For studies that sought to replicate previous findings, the percentage of null results was slightly higher, at 66%, whereas this figure stood at 55% for original research (see ‘Registered reports cut publication bias’).Matthew Warren, “First analysis of ‘pre-registered’ studies shows sharp rise in null findings” at Nature

It became easier to admit that one’s hypothesis “needs work” or however you want to put it.

It’s great to see somebody actually trying to do something about the long-standing file drawer problem. In general, the pace of reform in science publication seems, at times, to match the pace we see at the Vatican. But science publishing doesn’t have the excuse of antiquity…

Follow UD News at Twitter!

See also: What can a huge retractions database teach us? Overall, improved vigilance has slowed the trend, but key problems remain, including manipulated images. If a picture is worth a thousand words, that’s about three to five paragraphs of falsehood.

4 Replies to “Surprise: Science thrives when people can admit they didn’t prove something

  1. 1
    polistra says:

    Starting with the null is the first fault. Null hyp is an arbitrary and artificial goal.

    If we want meaningful and useful science, we should require meaningful and useful questions. Every study should start with a Real problem that needs a Real solution to make life better for Real people. If those three Reals aren’t there, it’s not worth the time and money.

  2. 2
    ScuzzaMan says:

    But that attitude shows poor collective reasoning skills. Showing which answers are wrong is, logically, a step on the road to right answers.

    There’s something wrong with this.

    Science is not an enterprise that is designed for, not capable of, finding right answers.

    Science is, a la Popper, an enterprise designed for, and only capable of, avoiding wrong answers.

    In parallel with the legal principle of presumption of innocence, science deals only in disproofs, since anyone with strong reasoning skills knows that proofs are not possible. Only increases in confidence by (A) lack of dis-proofs in overwhelming number, which includes as a subset (B) repeated experimental results.

    But it takes only a single counterfactual to disprove any proposition. The best we can say about any theory is that we have not disproved it after repeated attempts.

  3. 3
    Tom Robbins says:

    ScuzzaMan – I COMPLETELY AGREE – I love hard science topics – especially physics and Biology, but I agree with you and it is overlooked by so many! Science does not prove ANYTHING, but as you put it so well, only can increase the confidence that something is a reasonable interpretation of WHAT (NOT WHY) we can observe and reproduce in OUR REALITY. Think about it – we can talk all we want about a LAW of Gravity, weather Newtonian or Einsteins refinement. BUT, it only describes or predicts what we will see when this “force” in in play – but science NEVER explains what gravity really is, what a force is let alone WHY there are these predictable rules our universe seems to conform to. And, our arrogance in subjects of Origins is absolutley astounding – this is why all theories of origins have a shelf life – these are HUGE mysteries, billions to hundreds of billions of years in the past, why would we not think otherwise? It is just as easy to correct an epicyclic Ptolemaic model, actually more likely for us to come up with this kind of incorrect model than something that approximates reality that is so very remote compared to the span of a single lifetime. But anyway, you said it much more concise than I did….

  4. 4
    jdk says:

    Proof is for mathematics. Science builds well-verified theories, but its conclusions are always provisional to some extent, and capable of being revised, extended, or in fairly rare cases overturned.

    This is really not the point of the OP: the word “prove” in the title of the OP is misleading, I think.

Leave a Reply