Uncommon Descent Serving The Intelligent Design Community

Peer review: Not publishing social science studies with null results harms public discourse

arroba Email

It’s even worse in social sciences than in the hard sciences, apparently.

From Nature:

Researchers at Stanford University in California have now measured the extent of the problem, finding that most null results in a sample of social-science studies were never published. This publication bias may cause others to waste time repeating the work, or conceal failed attempts to replicate published research. Although already recognized as a problem, “it’s previously been hard to prove because unpublished results are hard to find”, says Stanford political scientist Neil Malhotra, who led the study.

His team investigated the fate of 221 sociological studies conducted between 2002 and 2012, which were recorded by Time-sharing Experiments for the Social Sciences (TESS), a US project that helps social scientists to carry out large-scale surveys of people’s views.


Of all the null studies, just 20% had appeared in a journal, and 65% had not even been written up. By contrast, roughly 60% of studies with strong results had been published.

From a lay perspective, this is a serious problem.

Take, for example, an inflammatory issue like racial, sexual, or religious discrimination. What if most of the studies that actually get published show that it exists in a given context—but studies that don’t support the thesis don’t usually get published (null result)?

Then people are likely to overestimate discrimination’s extent and influence in their lives. Indeed, they can point to “the published research” for support. But “the published research” is skewed specifically in favour of showing that the discrimination exists!

Access to the whole picture might show us a much more complex, muted reality, where discrimination is often a factor—but not the overwhelming factor some might fear.

Also, really bad or fraudulent papers are probably harder to spot in an environment that features a systematic skew:

On his return trip to Tilburg, Stapel stopped at the train station in Utrecht. This was the site of his study linking racism to environmental untidiness, supposedly conducted during a strike by sanitation workers. In the experiment described in the Science paper, white volunteers were invited to fill out a questionnaire in a seat among a row of six chairs; the row was empty except for the first chair, which was taken by a black occupant or a white one. Stapel and his co-author claimed that white volunteers tended to sit farther away from the black person when the surrounding area was strewn with garbage. Now, looking around during rush hour, as people streamed on and off the platforms, Stapel could not find a location that matched the conditions described in his experiment.

If the peer reviewers at Science had had the advantage of a wider range of actual results from social science (which might have featured more null result papers), they might have asked more questions in the first place.

– O’Leary for News

Follow UD News at Twitter!

Scientists' reputation suffered when a pair of stem cell studies had to be retracted. http://www.genengnews.com/gen-news-highlights/riken-center-to-reorganize-in-wake-of-stem-cell-scandal/81250291/ Dionisio
Unfortunately, this is true in almost any field. But is it because the peer review process weeds them out, or is it because the researches who obtain null results simply do not submit them for peer review? In my own limited experience, I have seen more of the latter. But, admittedly, this may be the response of past peer review experiences. Acartia_bogart

Leave a Reply