National Association of Scholars launches new report on the reproducibility crisis in science

_{Denyse O'Leary

April 27, 2018

Culture, Intelligent Design, Peer review, Science}

Share: Facebook; Twitter; LinkedIn; Flipboard; Print; Email

Report: NAS Launches New Report: “The Irreproducibility Crisis of Modern Science”:

Today is the launch of NAS’s newest report by David Randall and Christopher Welser. The Irreproducibility Crisis of Modern Science: Causes, Consequences, and the Road to Reform examines the different aspects of the reproducibility crisis of modern science. Our goal is to bring the reproducibility crisis to the forefront of public awareness and to call on policymakers to take effective steps to address it. We also include a series of policy recommendations, scientific and political, for alleviating the reproducibility crisis.

NAS was founded on, and continues to be guided by the idea that the pursuit of truth is the highest purpose of scholarly work. Civil and open debate is how we best pursue truth. We see this report as a first word on the crisis – not the last, and we wholeheartedly invite comments and responses to our report. More.

At the Report’s intro page:

A reproducibility crisis afflicts a wide range of scientific and social-scientific disciplines, from epidemiology to social psychology. Improper use of statistics, arbitrary research techniques, lack of accountability, political groupthink, and a scientific culture biased toward producing positive results together have produced a critical state of affairs. Many supposedly scientific results cannot be reproduced in subsequent investigations.

This study examines the different aspects of the reproducibility crisis of modern science. The report also includes a series of policy recommendations, scientific and political, for alleviating the reproducibility crisis. More.

From the Preface by Peter Wood:

The National Association of Scholars (NAS) has long been interested in the politicization of science. We have also long been interested in the search for truth—but mainly as it pertains to the humanities and social sciences. The irreproducibility crisis brings together our two long-time interests, because the inability of science to discern truth properly and its politicization go hand in hand.

Evergreen’s Bret Weinstein could testify to that last bit.

The cultural question we face today is whether we need science advances more or less than we need the imposition on science of post-modern cultural demands.

Note: Multiple pdfs via intro page but open access.

See also: Crisis in replication

and

At Quillette: Who will the Evergreen mob (targeted biology teacher recently) target next?

Comments

Excellent post @9 KF.tribune7_{May 8, 2018
May
05
May
8
08
2018
05:59 AM
5
05
59
AM
PDT}

AK, I see your:
If they are published in reputable peer reviewed journals, they are scientific findings.
we need to distinguish key terms and address underlying issues on logic and warrant. Truth (following Ari who got it right) says of what is that it is and of what is not that it is not -- accurate description of reality. As potentially knowing, rational and responsible subjects, we face the challenge that we are finite, fallible, morally struggling (is is not ought) and too often ill-willed. To credibly know objective truth we must move to warrant, that is knowledge in a relevant soft sense is warranted, credibly true and reliable belief. Belief highlighting that one or more subjects accepts it as so for good cause. Where, empirical claims are about the world we can experience, observe (and sometimes measure), in principle in common. And yes, I am using a soft sense of knowledge where there is an abstract possibility of error even in the face of credible conviction sufficient to responsibly ground acceptance and trust as reliable and accurate. In this context, facts are well grounded, relatively accessible reliable knowledge claims rooted in experience and/or observation. They are not privileged beyond that. Where, too, logic in large part is about the intellectual responsibilities of warrant, driven by certain first principles that are self-evident. For instance, at the heart of rationality is the distinction, A vs ~A. From distinct identity we instantly have the triple first principles of right reason, LOI {A is A, itself, rooted in its core characteristics that mark it as distinct, for, world W = {A|~A}]; LEM (any distinct x in the world W will be A or else ~A, not neither or both, i.e. A X-OR ~A); LNC (any x in W cannot be A AND ~A). This also founds mathematics through establishing 1-ness [A] and 2-ness, implying also 0-ness, thence the natural numbers etc and relevant structures. Math can be understood as the logic of structure and quantity. Math is pivotal, as it is essential to science but is exactly an abstract, rationally contemplative responsible discipline that astonishingly applies to the world of common experience. This is a big clue about our nature as rational, responsible, morally governed creatures living in a world where IS and OUGHT must be bridged at world-root level. Yes, first philosophy and linked issues are also highly relevant. Indeed the discussion of knowledge above is a manifestation of epistemology in action. That we are studying things in a world also points to the logic of being and many linked considerations such as possible vs impossible being, contingent and necessary being, rationality and intelligibility of existence [and non-existence], cause and effect etc. Going on, we are at the principle of grand coherence: reality exists all together, so all actually true descriptions are true together. That is, all actually true claims or descriptions will be true together. No two truths x and y can be such that y = ~x. This is a powerful test for extending and constraining the domain of knowledge. Now, too, we are speaking generally, there is here no privileging of any particular domain as authoritatively established knowledge that only a guild of the elites may warrant to be so. In principle, a little child may lead them in ANY domain of knowledge. That is, appeals to individual or collective expertise or authority are relativised. No authority -- including the Scientific Establishment (and its proxies, the editorial board of a sufficiently prestigious journal, conference or PhD examination committee, etc) -- is better than underlying facts, reasoning and assumptions. Including, worldview-level assumptions and ideological commitments. In short, there is no distinctive domain, science, which has a privileged body of knowledge arrived at using a method unique to that praxis. This is the sense where Feyerabend's anything goes is spot on. The issue in science is responsible warrant in empirical domains, not gatekeeper games. So, sciences and their methods will overlap considerably with other fields of responsible praxis. The school-level "scientific method" is more generic than is often given credit and it tends to under-emphasise the role of logic and epistemology as well as the delicate balance involved in peer review. As a start, who are the relevant peers, why? Those who dismissed the Semmelweis proposals on hand washing or those who overlooked the significance of a certain monk's studies of the inheritance patterns of peas? What about the difference between manipulated data in those studies and those on Burt's twins studies used to found educational policies? And much more. In short, the gatekeeper games, demarcation gambit on science vs non-science, pseudo-science and even anti-science is itself dubious. Let us instead adhere to responsible warrant. On which grounds, it is easy to see that there are indeed well founded reliable signs that certain entities are the result of intelligently directed configuration. That is backed by a trillion member observation base and by needle in haystack search analysis. In short, the design inference on observable signs is in fact legitimately and responsibly scientific. KFkairosfocus_{May 8, 2018
May
05
May
8
08
2018
01:27 AM
1
01
27
AM
PDT}

AK @ 5:
If they are published in reputable peer reviewed journals, they are scientific findings.They may be based on flawed assumptions, or fail to take into critical factors, or not be reproducible. But that doesn’t mean that they aren’t scientific.
Might be truth in practice, but it's terrible as an ideal. Firstly, we rest upon the subjectivity of "reputable" and "peer reviewed". This makes science socially defined, and out of reach from fact or reason outside of the herd. Also, it invites equivocations of "herd-science" with objective reasoning and truth. And irreproducibility crises. ET @ 6:
And yet such things are refuted, meaning they never were scientific findings.
I would want to soften this to "refuted by knowledge and reason available to the authors and peers" to allow efforts made in good faith and with due diligence to remain in consideration as scientific enterprises. It wouldn't spare the likely perpetrators of the bulk of this issue.LocalMinimum_{May 4, 2018
May
05
May
4
04
2018
10:23 AM
10
10
23
AM
PDT}

It may seems like a joke, but anyone familiar with the literature may know that parapsychology has some of the most rigorous standards in science for controlling the file drawer effect and ensuring quality in meta-analyses. Why is it so? Because it has been constantly under scrutiny and attack. Mainstream science could benefit from the same.xenos_{May 4, 2018
May
05
May
4
04
2018
04:46 AM
4
04
46
AM
PDT}

Allan:
If they are published in reputable peer reviewed journals, they are scientific findings.
And yet such things are refuted, meaning they never were scientific findings.
But that doesn’t mean that they aren’t scientific.
Yes, it does. If the research is false then it is not scientific.
You are falling into the same fallacy that ID proponents frequently accuse their opposition of. I.e, that if something is scientific that it is fact.
How is that a fallacy? Since when is science OK with falsehoods?
You are reading something into my comments that is not there.
It's there.
I was saying that there is very little incentive to repeat someone else’s work because doing so does not advance your career.
Right but by publishing unreproducible nonsense is OK.
Maybe this situation will improve when ID researchers start to publish their work in peer reviewed journals.
There is plenty of evidence for ID already in peer-review. And there isn't anything in peer-review that supports evolutionism, ie evolution by means of blind and mindless processes. You arte clearly confused.ET_{April 28, 2018
April
04
Apr
28
28
2018
11:03 AM
11
11
03
AM
PDT}

ET,
Question-begging- how does anyone know whether or not they are scientific findings?
If they are published in reputable peer reviewed journals, they are scientific findings. They may be based on flawed assumptions, or fail to take into critical factors, or not be reproducible. But that doesn’t mean that they aren’t scientific. You are falling into the same fallacy that ID proponents frequently accuse their opposition of. I.e, that if something is scientific that it is fact.
But it’s OK to get tenure by publishing nonsense? Really?
You are reading something into my comments that is not there. A common affliction of those with an obsessive hatred of a/mats. I was saying that there is very little incentive to repeat someone else’s work because doing so does not advance your career. I didn’t say that this was acceptable. I was just making a statement of fact. Maybe this situation will improve when ID researchers start to publish their work in peer reviewed journals.Allan Keith_{April 28, 2018
April
04
Apr
28
28
2018
07:30 AM
7
07
30
AM
PDT}

The cause is theory. The cure is ZERO THEORY.polistra_{April 27, 2018
April
04
Apr
27
27
2018
11:29 PM
11
11
29
PM
PDT}

Allan:
I think a bigger problem is the fact that attempts to reproduce scientific findings are rare in many fields.
Question-begging- how does anyone know whether or not they are scientific findings?
You don’t get tenure by reproducing someone else’s work.
But it's OK to get tenure by publishing nonsense? Really?ET_{April 27, 2018
April
04
Apr
27
27
2018
05:17 PM
5
05
17
PM
PDT}

The National Association of Scholars (NAS) has long been interested in the politicization of science.
Very good.
Would it surprise you to learn the greatest global two-year cooling event of the last century just occurred? From February 2016 to February 2018 (the latest month available) global average temperatures dropped 0.56°C. You have to go back to 1982-84 for the next biggest two-year drop, 0.47°C—also during the global warming era. All the data in this essay come from GISTEMP Team, 2018: GISS Surface Temperature Analysis (GISTEMP). NASA Goddard Institute for Space Studies (dataset accessed 2018-04-11 at NASA). [Aron Brown, RealClearMarkets.com]
Origenes_{April 27, 2018
April
04
Apr
27
27
2018
04:11 PM
4
04
11
PM
PDT}

I think a bigger problem is the fact that attempts to reproduce scientific findings are rare in many fields. You don’t get tenure by reproducing someone else’s work.Allan Keith_{April 27, 2018
April
04
Apr
27
27
2018
03:47 PM
3
03
47
PM
PDT}

You must be logged in to post a comment.

Leave a Reply