Measuring the effectiveness of scientific gatekeeping
Significance Peer review is an institution of enormous importance for the careers of scientists and the content of published science. The decisions of gatekeepers—editors and peer reviewers—legitimize scientific findings, distribute professional rewards, and influence future research. However, appropriate data to gauge the quality of gatekeeper decision-making in science has rarely been made publicly available. Our research tracks the popularity of rejected and accepted manuscripts at three elite medical journals. We found that editors and reviewers generally made good decisions regarding which manuscripts to promote and reject. However, many highly cited articles were surprisingly rejected. Our research suggests that evaluative strategies that increase the mean quality of published science may also increase the risk of rejecting unconventional or outstanding work. (paywall) – Kyle Silera, Kirby Leeb, and Lisa Beroc, December 22, 2014, doi: 10.1073/pnas.1418218112 PNAS January 13, 2015 vol. 112 no. 2 360-365
Rob Sheldon explains it thus:
I think this paper strengthens Kuhn’s thesis that science is very good at sifting mediocre papers near the center of the distribution. It is just plain lousy at recognizing the upper tail of the distribution, and being risk-averse, lumps it with the rejected lower tail.
This means a field advances not by quantum leaps of Levy flight, but at the slow diffusive rate of a somnolent dinosaur, interspersed with meteoritic paradigm craters.
See also: If peer review is working, why all the retractions?
Visit Retraction Watch for up to date news on and honest evaluation of the peer review system.
Follow UD News at Twitter!