In “Replication studies: Bad copy” (Nature, 16 May 2012), Ed Yong reports, “In the wake of high-profile controversies, psychologists are facing up to problems with replication.”:
Positive results in psychology can behave like rumours: easy to release but hard to dispel. They dominate most journals, which strive to present new, exciting research. Meanwhile, attempts to replicate those studies, especially when the findings are negative, go unpublished, languishing in personal file drawers or circulating in conversations around the water cooler. “There are some experiments that everyone knows don’t replicate, but this knowledge doesn’t get into the literature,” says Wagenmakers. The publication barrier can be chilling, he adds. “I’ve seen students spending their entire PhD period trying to replicate a phenomenon, failing, and quitting academia because they had nothing to show for their time.”
These problems occur throughout the sciences, but psychology has a number of deeply entrenched cultural norms that exacerbate them. It has become common practice, for example, to tweak experimental designs in ways that practically guarantee positive results. And once positive results are published, few researchers replicate the experiment exactly, instead carrying out ‘conceptual replications’ that test similar hypotheses using different methods. This practice, say critics, builds a house of cards on potentially shaky foundations.
Of course, cleaning house at those journals would be a darn good idea, but let’s not break out the bubbly yet.
The fad for replication could prove to be nothing more than an opportunity to stifle research that is unpopular anyway, while popular but questionable research is somehow “replicated,” because the same tricks, outlined in the Nature article, are played.
Key take-home point:
Psychology and psychiatry, according to other work by Fanelli, are the worst offenders: they are five times more likely to report a positive result than are the space sciences, which are at the other end of the spectrum (see ‘Accentuate the positive’). The situation is not improving. In 1959, statistician Theodore Sterling found that 97% of the studies in four major psychology journals had reported statistically significant positive results. When he repeated the analysis in 1995, nothing had changed.
Well, the space sciences are kept honest by the fact that there really aren’t aliens out there constantly jiggering the results.
So isn’t the problem here that psychology and psychiatry shouldn’t be considered sciences at all?
There is nothing wrong with a discipline not being considered a science. A good fundraiser for cancer research practises a skill that doesn’t resolve to a science. But that doesn’t make it any less important to cancer research.
Or even any less exact. After all, she is judged by the dollars and cents amount that she raises, which – you must admit – is a pretty exact measure.
Thoughts?
Hat tip: Stephanie West Allan at Brains on Purpose
Follow UD News at Twitter!