Peer review

P threshold values often likely false?

Spread the love

From John Ioannidis at the Journal of the American Medical Association:

vPalues and accompanying methods of statistical significance testing are creating challenges in biomedical science and other disciplines. The vast majority (96%) of articles that report P values in the abstract, full text, or both include some values of .05 or less. However, many of the claims that these reports highlight are likely false. Recognizing the major importance of the statistical significance conundrum, the American Statistical Association (ASA) published3 a statement on P values in 2016. More.

See also: P-values: Scientists slam proposal to raise threshold for statistically significant findings

and

Ioannidis again on misleading meta-analyses

4 Replies to “P threshold values often likely false?

  1. 1
  2. 2
    polistra says:

    Just get rid of stats.

    A clear and meaningful observation is clear and meaningful by itself, without stats.

    If you can’t set up an experiment or observation that yields a clear decision without stats, you still have a useful result. The fact that you CAN’T set up such an experiment IS a useful piece of information and deserves to be reported.

    If you have to use stats of any kind, you’re fudging and faking by definition.

  3. 3
    Bob O'H says:

    polistra – OK, I’m a statistician so I might be biased, but I don’t think you appreciate the impossibility of doing science without statistics. For example, how would you know if a new drug worked or was actually harmful? How would you design an experiment to see if a drug could cure the common cold? Remember that the severity of a cold varies, and this depends on, amongst other things, the age and health of the person infected)? Also remember that any bad side effects are unlikely to show themselves in all patients.

  4. 4
    gpuccio says:

    Bob O’H:

    I work with statistics in medicine, and of course I agree with you.

    It’s true that there is a lot of abuse of statistics, and the main reason is that almost no one understands or know it, but practically everybody has to use it.

    Regarding p values, my idea is simple: it’s note a problem of setting thresholds, o.o5 or o.o1.

    The problem is that there should be no absolute thersholds, but simply a credible assessment of the results, case by case.

    It has been my experience that in some cases, where the results were significant at a level of <10^-16 (the lowest value computed by R), a reviewer asked that the p value were reported simply as <0.01.

    That's, IMO, simple folly. We need to understand what a p value means in its specific contex, and not just classify the results as "significant" or "not significant".

Leave a Reply