It’s even worse in social sciences than in the hard sciences, apparently.
From Nature:
Researchers at Stanford University in California have now measured the extent of the problem, finding that most null results in a sample of social-science studies were never published. This publication bias may cause others to waste time repeating the work, or conceal failed attempts to replicate published research. Although already recognized as a problem, “it’s previously been hard to prove because unpublished results are hard to find”, says Stanford political scientist Neil Malhotra, who led the study.
His team investigated the fate of 221 sociological studies conducted between 2002 and 2012, which were recorded by Time-sharing Experiments for the Social Sciences (TESS), a US project that helps social scientists to carry out large-scale surveys of people’s views.
And
Of all the null studies, just 20% had appeared in a journal, and 65% had not even been written up. By contrast, roughly 60% of studies with strong results had been published.
From a lay perspective, this is a serious problem.
Take, for example, an inflammatory issue like racial, sexual, or religious discrimination. What if most of the studies that actually get published show that it exists in a given context—but studies that don’t support the thesis don’t usually get published (null result)?
Then people are likely to overestimate discrimination’s extent and influence in their lives. Indeed, they can point to “the published research” for support. But “the published research” is skewed specifically in favour of showing that the discrimination exists!
Access to the whole picture might show us a much more complex, muted reality, where discrimination is often a factor—but not the overwhelming factor some might fear.
Also, really bad or fraudulent papers are probably harder to spot in an environment that features a systematic skew:
On his return trip to Tilburg, Stapel stopped at the train station in Utrecht. This was the site of his study linking racism to environmental untidiness, supposedly conducted during a strike by sanitation workers. In the experiment described in the Science paper, white volunteers were invited to fill out a questionnaire in a seat among a row of six chairs; the row was empty except for the first chair, which was taken by a black occupant or a white one. Stapel and his co-author claimed that white volunteers tended to sit farther away from the black person when the surrounding area was strewn with garbage. Now, looking around during rush hour, as people streamed on and off the platforms, Stapel could not find a location that matched the conditions described in his experiment.
If the peer reviewers at Science had had the advantage of a wider range of actual results from social science (which might have featured more null result papers), they might have asked more questions in the first place.
– O’Leary for News
Follow UD News at Twitter!