Peer review Psychology

Half of social science replication studies failed under near-ideal conditions

Spread the love
File:FileStack.jpg
What’s hot? What’s not?/Niklas Bildhauer, Wikimedia

It’s becoming harder to ignore the stench:

The drive recruited labs around the world to try to replicate the results of 28 classic and contemporary psychology experiments. Only half were reproduced successfully using a strict threshold for significance that was set at P < 0.0001 (the P value is a common test for judging the strength of scientific evidence). Brian Owens, “Replication failures in psychology not due to differences in study populations” at Nature

So why did so many classic studies fail? The account in Nature doesn’t say but that won’t The Atlantic, stop people wondering:

Despite the large sample sizes and the blessings of the original teams, the team failed to replicate half of the studies it focused on. It couldn’t, for example, show that people subconsciously exposed to the concept of heat were more likely to believe in global warming, or that moral transgressions create a need for physical cleanliness in the style of Lady Macbeth, or that people who grow up with more siblings are more altruistic. And as in previous big projects, online bettors were surprisingly good at predicting beforehand which studies would ultimately replicate. Somehow, they could intuit which studies were reliable. Ed Yong, “Psychology’s Replication Crisis Is Running Out of Excuses” at The Atlantic

Someone should interview the online bettors about how they knew which studies would replicate and which to avoid. Were the latter studies designed to demonstrate some new hotness or Cool that would play well at TED talks?

Follow UD News at Twitter!

See also: Why do we think “social psychology” is science anyway?

At the New York Times: Defending the failures of social science as “science”

What’s wrong with social psychology, in a nutshell

How political bias affects social science research

Stanford Prison Experiment findings a “sham” – but how much of social psychology is legitimate anyway?

BS detector for the social sciences

and

All sides agree: progressive politics is strangling social sciences

4 Replies to “Half of social science replication studies failed under near-ideal conditions

  1. 1
    aarceng says:

    Isn’t P < 0.0001 rather extreme compared the more commonly accepted P < 0.05? I would expect a large number would fail to replicate under those conditions.

  2. 2
    AaronS1978 says:

    Do these failed replications include the undiably true result provided by evo psychology?

  3. 3
    Bob O'H says:

    aarceng – yes, that was my reaction as well. With that threshold, no wonder so many fail!

  4. 4
    Ed George says:

    aarceng and Bob O’H, with that threshold I am surprised that 50% were replicable. What is that? A 99.99% confidence level? As an analytical chemist, when I calibrate an ICP, a very accurate and precise instrument, an r squared of 0.9999 is almost unheard of. More reasonable would be an r squared of 0.995 or, on a really good day, 0.999.

    If half of the studies were repeatable at that level of confidence, the field of social science can hold its head high.

Leave a Reply