We don’t often say “be sure to read this article,” but be sure to read this article by Jerry Adler in Pacific Standard, on the background to many recent instances of research misconduct.
Something unprecedented has occurred in the last couple of decades in the social sciences. Overlaid on the usual academic incentives of tenure, advancement, grants, and prizes are the glittering rewards of celebrity, best-selling books, magazine profiles, TED talks, and TV appearances. A whole industry has grown up around marketing the surprising-yet-oddly-intuitive findings of social psychology, behavioral economics, and related fields. The success of authors who popularize academic work—Malcolm Gladwell, the Freakonomics guys, and the now-disgraced Jonah Lehrer—has stoked an enormous appetite for usable wisdom from the social sciences. And the whole ecosystem feeds on new, dramatic findings from the lab. “We are living in an age that glorifies the single study,” says Nina Strohminger, a Duke post-doc in social psychology. “It’s a folly perpetuated not just by scientists, but by academic journals, the media, granting agencies—we’re all complicit in this hunger for fast, definitive answers.”
Yes, and—though Adler never quite gets here—very often, complicit in the hunger for answers that confirm what people think they already know. Not only that, an honest study that came up with genuinely politically incorrect findings might be banished on principle, along with the researcher.
The researcher is supposed to realize that only politically correct findings are part of the quest for knowledge.
Adler’s article addresses how p-values are misused to produce the desired results:
P is a central concept in statistics: It’s the mathematical factor that mediates between what happens in the laboratory and what happens in the real world. The most common form of statistical analysis proceeds by a kind of backwards logic: Technically, the researcher is trying to disprove the “null hypothesis,” the assumption that the condition under investigation actually makes no difference. In Sanna’s experiment, the null hypothesis is that elevation has no effect on how much hot sauce people dole out. If that is actually what the data shows, then the experiment is over, the null hypothesis wins, and the researcher can forget about going on The Daily Show. But in practice things usually aren’t quite as clear-cut.
In psychology, p value is less than .05 or five percent. For the Higgs boson, it was one in 3 million.
As Simmons showed, psychologists who deploy enough statistical sleight of hand can find “significance” in almost any data set. How often do researchers give in to this temptation? One way to roughly answer that question would be to study the distribution of p values over a large sample of papers. If researchers are fiddling with the math to get their results just under the 0.05 threshold, then you might expect to see a cluster of values just below 0.05, rather than the more normal distribution that might have arisen as a result of chance. In an analysis of a year’s worth of papers in three leading psychology journals, two researchers found “a peculiar prevalence of p values just below 0.05.”
Adler discusses current attempts at reform, and one can only wish the reformers well. The difficulty is that in a world where nature may or may not co-operate with the “correct” results, the problems may well lie deeper than the intended reforms.
Hat tip: Stephanie West Allen at Brains on Purpose
Follow UD News at Twitter!