Does reliability of published works decrease with journal rank?

_{Denyse O'Leary

February 22, 2018

Intelligent Design, Peer review}

Share: Facebook; Twitter; LinkedIn; Flipboard; Print; Email

From neuroscientist (zoology) Björn Brembs at Frontier in Human Neuroscience:

In which journal a scientist publishes is considered one of the most crucial factors determining their career. The underlying common assumption is that only the best scientists manage to publish in a highly selective tier of the most prestigious journals. However, data from several lines of evidence suggest that the methodological quality of scientific experiments does not increase with increasing rank of the journal. On the contrary, an accumulating body of evidence suggests the inverse: methodological quality and, consequently, reliability of published research works in several fields may be decreasing with increasing journal rank. The data supporting these conclusions circumvent confounding factors such as increased readership and scrutiny for these journals, focusing instead on quantifiable indicators of methodological soundness in the published literature, relying on, in part, semi-automated data extraction from often thousands of publications at a time. With the accumulating evidence over the last decade grew the realization that the very existence of scholarly journals, due to their inherent hierarchy, constitutes one of the major threats to publicly funded science: hiring, promoting and funding scientists who publish unreliable science eventually erodes public trust in science. – Front. Hum. Neurosci., 20 February 2018 | https://doi.org/10.3389/fnhum.2018.00037More.

File:FileStack.jpg — What’s hot? What’s not?/Niklas Bildhauer, Wikimedia

We should wonder why this stuff isn’t a way bigger scandal. That’s what unthinking piety does to a field, and unthinking piety toward science is what much of the current popular science media promotes.

See also: Missing data hinder replication
in AI studies too?
At Nautilus: Scientists should not accept unreplicated results
– yawn

At Nature: Change how we judge research. Hmm… Using this scheme, what would protect the researcher who submits the suggested bio-sketch from becoming a target for political reasons that are unrelated to research quality? Think Jordan B. Peterson. or Gunter Bechly. Or anyone who sounds like a risk for blowing the whistle on corruption. The fate of whistleblowers is already often grim.

and

Does it matter in science if no one can replicate your results?

Comments

Here's a paper that should get some attention:
An incentive structure that rewards publication quantity will, in the absence of countervailing forces, select for methods that produce the greatest number of publishable results. This, in turn, will lead to the natural selection of poor methods and increasingly high false discovery rates. . . . If shallower work generating more publications is favoured, then researchers interested in pursuing complex questions may find themselves without jobs, perhaps to the detriment of the scientific community more broadly.
PaV

I think the only scandal is the amount of status we give to academia in general. The idea that "peer review is gold standard" nonsense is ridiculous. If we just instead treated all of these as basic professional journals, and the people in them having jobs just like everyone else, and the people publishing in them having prejudices just like everyone else, then there would be very little at issue. The reason why there is an issue is that people pretend that something in "science" must necessarily be true. It is only because the journals set themselves up as arbiters of truth, and because the public at large goes along with this charade, that this continues. I think that rigor is great. But everything must be taken with a grain of salt. Giving undue weight to special journals, and too little weight to outliers, is just asking to be hoodwinked. How could it not be? Unfortunately, the entire media runs on status. Newspapers only report science from "reputable" journals; they only report ideas that have a "good reputation"; etc. Our respectability is killing us, and is actually making us disrespectable. There is a phobia right now about being wrong, and part of that is because everyone wants people to think that they are always right. Why can't we be okay with being wrong every once in a while? Why is the possibility of being wrong in a "non-respectable" direction so problematic to the modern mind? What this phobia about being wrong has done is actually solidified certain types of wrongness in such a way as to make them unquestionable. It's solidified some journals in such a way as to make them "authoritative" and almost beyond question. None of this is good for academics. johnnyb

You must be logged in to post a comment.

Leave a Reply