Here’s Jelte M. Wicherts in Nature (30 November 2011) on how “Psychology must learn a lesson from fraud case,” (the Stapel affair) suggesting that sharing data might be the answer:
In a 2006 study published in American Psychologist, I helped to show that almost three-quarters of researchers who had published a paper in a high-impact psychology journal had not shared their data (J. M. Wicherts et al. Am. Psychol. 61, 726–728; 2006). Several data sets, authors said, had been misplaced, whereas others were kept secret because they were part of ongoing work, or because of ethical rules meant to protect participants’ privacy. Such confidentiality has long been the most common excuse that psychologists offer for not sharing data, but in practice, most simply fail to document their data in a way that allows others to quickly and easily check their work. It is not unusual for data that are shared to list variables only as VAR00001 through VAR00019, with no further explanation.
It is not just misconduct that flourishes in such secrecy. So too do the common and more insidious failings of error and bias in data analysis — for example, the use of incorrect tests, reporting errors due to similarly named variables, favouring results that confirm a hypothesis and overly positive reporting of statistical outcomes.
Psychology’s culture of secrecy produces substandard science. Reanalyses of statistics in published psychology papers show frequent errors, and the more reluctant authors are to share their data, the more likely it is that their papers will contain mistakes. Or to put it another way — the results that most need checking cannot be checked.
Wicherts’ proposal, obligatory archiving of raw data, does not come at the central problem: Psychology’s addiction to politically correct findings and – some would argue – surprising findings. In other words, if most psychologists feel the need to see a situation in a certain way, showing them the raw data may only produce a resounding chorus of yesses to some very flimsy interpretation.
As for “surprising” data: We suggest that, in general, it is accepted if it reinforces political correctness and does not pose a threat to consensus. Otherwise, if published, it falls down the memory hole.
One obvious solution is to quit pretending that psychology is a science. No discipline that depends much on political correctness can function as a science anyway.