Uncommon Descent Serving The Intelligent Design Community

Robert J. Marks: Reforming peer review faces serious numerical law problems

What’s hot? What’s not?/Niklas Bildhauer, Wikimedia

Laws concerning the way people behave around numbers mean that quantification itself invites certain types of corruption:

Goodhart’s Law, for example, captures the unintended effect of using numerical metrics as goals: “When a measure becomes a target, it ceases to be a good measure.”1

British economist Charles Goodhart’s law applies well beyond his specialty. As a general principle, focusing exclusively on a measurable numerical goal distracts from achieving the broader outcome that the metric is trying to track. Players focus on the numbers and try to game them, which often leads to deception and dishonesty. Crossing the finish line becomes more important than how you got there.

A consequence of Goodhart’s Law is the temptation to cheat to achieve a goal. This is called Campbell’s Law, after an American social scientist, Donald T. Campbell: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

In other words, measurement creates a temptation to achieve a measurable goal by less than totally honest means. As in physics, the simple act of measuring invariably disturbs what you are trying to measure.

As a professor at research universities for over forty years, I have observed that both Goodhart’s Law and Campbell’s Law have had a big impact in higher academia. Much of the impact has been negative.Robert J. Marks, “Why it is so hard to reform peer review” at Mind Matters News

It can be fought, he says, but there are no quick fixes.

Also by Robert J. Marks: Study shows that eating raisins causes plantar warts. Sure. Because, if you torture a Big Data enough, it will confess to anything


Things exist that are unknowable: A tutorial on Chaitin’s number

Follow UD News at Twitter!


Leave a Reply