Intelligent Design

Why “statistical significance” is corrupting science

Spread the love

Because, as Pomona College prof Gary Smith points out, computers excel at finding temporary patterns and that tendency contributes to the replication crisis in science:

When I was a young assistant professor at Yale, one of my senior colleagues, Nobel Laureate James Tobin, wryly observed that the bad old days when researchers had to do calculations by hand were actually a blessing. The effort was so great that people thought hard before calculating. They put theory before data. Today, with terabytes of data and lightning-fast computers, it is too easy to calculate first, think later.

Ironically, a Yale professor named Aleh Tsyvinski recently went on a massive data mining expedition with Yukun Liu (then a graduate student, now a professor himself at the University of Rochester). Even more ironically, Tsyvinski is the Arthur M. Okun Professor of Economics at Yale, an endowed chair named after Tobin’s close friend Arthur Okun. Okun was a consummate believer in economic analysis that makes sense and is useful.

Tsyvinski and Liu set out to find statistical correlations between bitcoin returns and other variables. There is no logical reason for bitcoin returns to be affected by anything other than guesses about future bitcoin returns. Unlike bonds that yield interest, stocks that yield dividends, apartments that yield rent, businesses that yield profits, and other real investments, bitcoin doesn’t yield anything at all, so there is no compelling way to value bitcoin the way investors can value bonds, stocks, apartments, businesses, and other real investments.

Attempts to correlate bitcoin returns with real economic variables are destined to disappoint. Yet Liu and Tsyvinski did exactly that and their disappointments are instructive.

Gary Smith, “Computers excel at finding temporary patterns” at Mind Matters News

Substitute “pterosaur” for “bitcoin” and you likely get the same problem in biology.

See also: Why it’s so hard to reform peer review. Reformers are battling numerical laws that govern how incentives work. Know your enemy! (Robert J. Marks)

30 Replies to “Why “statistical significance” is corrupting science

  1. 1
    polistra says:

    If you have to use stats you’re not doing science.

    A properly designed experiment or observation will show the variable, or the constant, clearly enough to avoid post-processing. If it doesn’t, you need to try smarter experiments or sharper observations. Using stats short-circuits the necessary motivation toward improved REAL science.

  2. 2
    Truthfreedom says:

    Today, with terabytes of data and lightning-fast computers, it is too easy to calculate first, think later.

    Oh my. Weren’t computers conscious according to our materialist friends? Don’t they think for themselves?

  3. 3
    Mac McTavish says:

    P

    If you have to use stats you’re not doing science.

    This has to be one of the most ridiculous statements I have ever read, that wasn’t written by ET. Statistics can, and often is, used improperly. But it is an invaluable tool in almost all scientific disciplines.

  4. 4
    ET says:

    LoL! @ Mac- read your own comments. Yours have become some of the most ridiculous posts ever. On the other hand all you can do is hurl false accusations at me.

    Pathetic, actually. As is your quote-mine of Polistra.

  5. 5
    Bob O'H says:

    If you have to use stats you’re not doing science.

    A properly designed experiment or observation will show the variable, or the constant, clearly enough to avoid post-processing.

    Never mind that the design of experiments is statistics. Or that when you are looking at the real world, you can’t always do experiments (e.g. in oil exploration). And even if you can, it’s impractical or unethical to collect enough data to not need statistics (e.g. clinical trials for Covid-19 vaccines).

  6. 6
    Truthfreedom says:

    Bob O’H

    Or that when you are looking at the real world …

    Are you an epistemological realist?

  7. 7
    Truthfreedom says:

    Calling time on ‘statistical significance’ in science research.

    Scientists should stop using the term ‘statistically significant’ in their research, urges this editorial in a special issue of The American Statistician published today.

    The issue, Statistical Inference in the 21st Century: A World Beyond P<0.05, calls for an end to the practice of using a probability value (p-value) of less than 0.05 as strong evidence against a null hypothesis or a value greater than 0.05 as strong evidence favoring a null hypothesis. Instead, p-values should be reported as continuous quantities and described in language stating what the value means in the scientific context.

    Containing 43 papers by statisticians from around the world, the special issue is expected to lead to a major rethinking of statistical inference by initiating a process that ultimately moves statistical science – and science — into a new age.
    Death Blow to “Statistical Significance”

    Maybe Bob O’H is one of the 43 signers?

  8. 8
    Bob O'H says:

    Tf – no I wasn’t, I wasn’t invited to join the special issue. To be honest, that special issue looks like it’s preaching to the converted.

  9. 9
    Truthfreedom says:

    8 Bob O’H

    That special issue looks like it’s preaching to the converted.

    Have you read it?

  10. 10
    Bob O'H says:

    Not all of it, but from what I have read of it and looking at the names and topics, and because I know the discussions that have gone on, I’m sure that the themes are ones that have been repeated in the statistical community for years.

  11. 11
    Truthfreedom says:

    10 Bob O’H

    Not all of it,

    To criticize a paper, you need to read it in its entirety. 🙂
    Can you imagine people criticizing the “theory of evolution” without knowing what it is?

  12. 12
    Bob O'H says:

    Can you imagine people criticizing the “theory of evolution” without knowing what it is?

    I don’t need to. I visit this site regularly.

  13. 13

    .
    Bob, is it somehow invalid and critical of evolution to say that it has symbolic physical requirements in order to come into being, and that these requirements were predicted and confirmed via experiment?

  14. 14
    Truthfreedom says:

    12 Bob O’H

    I don’t need to. I visit this site regularly.

    But you criticize people for (supposedly) doing what you are doing.

  15. 15
    Bob O'H says:

    UB @ 13 – it’s certainly invalid to say something that doesn’t make any sense. Even if I knew what symbolic physical requirements were, I couldn’t comment because you don’t bother to give any more details.

    Tf @ 14 – I’m a professional statistician. I have followed these discussions, and played a (very minor) role in some of them. I’m confident I have the expertise and knowledge to be able to evaluate the broad thrust of the special issue from the titles, and author names. Certainly enough to know that the main messages are one that most statisticians will already be aware of.

  16. 16
    Truthfreedom says:

    Bob O’H

    I have the expertise and knowledge to be able to evaluate the broad thrust of the special issue from the titles, and author names.

    No knowledge of a paper you have not read in its entirety. You can’t know its content until you have finished it (unless you are trying to convince me that you have psychic powers).

  17. 17
    Truthfreedom says:

    15 Bob O’H

    It’s certainly invalid to say something that doesn’t make any sense.

    What would you say of a person who says he believes both at the same time in epistemological realism AND epistemological idealism? (He practices both).

  18. 18
    Bob O'H says:

    Tf – I certainly can’t know all of the contents. But if a paper is titled “A proof of e = mc^2”, I can be fairly sure it’s a proof of e = mc^2. And if it’s by R.B. O’Hara of NTNU, I’d be on pretty firm ground in concluding it was probably rubbish.

  19. 19

    .
    Facepalm

    So Bob, when you conjured up distributive system theory to argue that that the gene system could be replicated without “anything immaterial” being required in the replication, you had no idea what the issues were?

    Bullshit. Deceptive bullshit.

  20. 20
    Truthfreedom says:

    18 Bob O’H

    Tf– I certainly can’t know all of the contents.

    Then you can’t judge them as “being preached to the choir”.
    It would be very bad if you were to partake in “peer review” studies: “Look, I have not read this paper but I think such-and-such…”
    Maybe that is one of the reasons why “peer review” is in trouble? Human folly and corruption?

  21. 21

    .
    Oh and Bob … just one more thing. Anytime you want to explore additional details of the semiotic argument (beyond those that you are already familiar with) just let me know. I’m happy to oblige. Perhaps we can start off with the distinction between a transcribable memory token and one that is not transcribable. The difference is in the constraints that establish the medium, Bob. One requires a special hierarchy in the constraints, and the other doesn’t. It’s all very fascinating that physicists have documented all this over the course of the past five or six decades, specifically with the origin of life in mind.

    😉

  22. 22
    Bob O'H says:

    Tf – I am heavily involved in the peer review system. When I have to provide a full review, I do read the whole paper, usually more than once. In this instance, it wasn’t necessary, because I only wanted to get a feel for the sorts of arguments being made. From that I came to a tentative conclusion about the contents, and reported what it looked like (to me, clearly).

  23. 23
    Bob O'H says:

    UB – if I want to discuss the semiotic argument, I’ll do it with someone who wants to actually discuss. For, example by explaining what they mean when they use jargon.

  24. 24

    .
    A powerful retort Bob. Powerful.

    In many ways, much better than “distributive systems theory”.

  25. 25
    kairosfocus says:

    BO’H:

    It is interesting to ask, why, after 20 years of advances in molecular biology and genetics from the 50’s – 70’s, the manifest import of the cumulative discoveries has been sidelined.

    We have identified, in the heart of the cell, alphanumeric, algorithmic — linguistic, goal-directed — code with complex, integrated execution machinery using molecular nanotech. Those are actually decisive signs that the cell is a designed entity, we have found our SETI signal in our own bodies.

    What then becomes interesting is how that is suppressed.

    Now, your answer is _______

    KF

  26. 26
    Truthfreedom says:

    22 Bob O’H

    From that I came to a tentative conclusion about the contents, and reported what it looked like (to me, clearly).

    That’s nice but it only reinforces my position of you being dogmatic and dismissive of other people’s work and ideas without bothering to know them. Which is the antithesis of the scientific mindset.
    Clearly? Clearly the Earth is flat!

  27. 27
    Bob O'H says:

    Tf – rather, I know them well enough to have a good idea about what they are going to say. It’s the sort of heuristic everyone uses.

  28. 28
    ET says:

    LoL! @ Bob O’H- buy a dictionary and learn how to use it.

    Bob O’H doesn’t know what the theory of evolution says. No one does. That’s because it doesn’t exist. Bob O’H is proud of his dishonesty.

  29. 29
    Truthfreedom says:

    27 Bob O’H

    Tf – rather, I know them well enough to have a good idea about what they are going to say.

    So dogmatic and mind reader.
    You are criticizing a published paper without reading it.

    It’s the sort of heuristic everyone uses.

    Everyone that is dogmatic and who wants to remain ignorant, not everyone in the strict sense.
    I wouldn’t dare to criticize a paper without finishing it. I may commit mistakes interpreting it, but I would finish it. 🙂
    You don’t and that’s the difference.

  30. 30

    .
    … The thinking man, lol.

    Bob: we can replicate the gene system without anything immaterial being added.

    UB: Bob, you’ll have to coordinate the multiple descriptions of decoding with all the descriptions that have to be decoded, otherwise the system cannot even begin to function. Where will that all simultaneous coordination come from?

    Bob: Distributive systems theory. We already know that works.

    UB: Wha?

    Bob: Yep, we get fresh papaya in the middle of winter in Norway. Proof!

    Brilliant!

Leave a Reply