In the recent social sciences scandals, there was an obvious “freakonomics” factor: Really weird findings that do not directly upset elite pieties get massive attention and little analysis. Now, in “Freakonomics: What Went Wrong?” (American Scientist, statistics teachers Andrew Gelman and Kaiser Fung explain, “Examination of a very popular popular-statistics series reveals avoidable errors”:
In our analysis of the Freakonomics approach, we encountered a range of avoidable mistakes, from back-of-the-envelope analyses gone wrong to unexamined assumptions to an uncritical reliance on the work of Levitt’s friends and colleagues. This turns accessibility on its head: Readers must work to discern which conclusions are fully quantitative, which are somewhat data driven and which are purely speculative.
The risks of driving a car: In SuperFreakonomics, Levitt and Dubner use a back-of-the-envelope calculation to make the contrarian claim that driving drunk is safer than walking drunk, an oversimplified argument that was picked apart by bloggers. The problem with this argument, and others like it, lies in the assumption that the driver and the walker are the same type of person, making the same kinds of choices, except for their choice of transportation. Such all-else-equal thinking is a common statistical fallacy. In fact, driver and walker are likely to differ in many ways other than their mode of travel. What seem like natural calculations are stymied by the impracticality, in real life, of changing one variable while leaving all other variables constant.
Some good suggestions for avoiding stats scams follow.