Tabby’s Star — on the “extraordinary evidence” claim

Headlined: https://uncommondesc.wpengine.com/mathematics/tabbys-star-3-the-business-of-dealing-with-black-swans/ kairosfocus

F/N: Let me clip Barsch as a public service for those dipping a tentative toe in the frigid, shark-infested waters of a chaotic black swan-prone environment:

Statisticians lament how few business managers think probabilistically. In a world awash with data, statisticians claim there are few reasons to not have a decent amount of objective data for decision making. However, there are some events for which there are no data (they haven’t occurred yet), and there are other events that could happen outside the scope of what we think is possible. The best quote to sum up this framework for decision making comes from the former US Defense secretary Donald Rumsfeld in February 2002: “There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – there are things we do not know we don’t know.” Breaking this statement down, it appears Mr. Rumsfeld is speaking about Frequentism, subjective probability (Bayes) and those rare but extreme events coined by Nassim Taleb as “Black Swans”. . . . . Rumsfeld seems to be saying we can guess the probability of the “known knowns” because they’ve happened before and we have frequent data to support objective reasoning. These “known knowns” are Nassim Taleb’s White Swans. There are also “known unknowns” or things that have never happened before, but have entered our imaginations as possible events (Taleb’s Grey Swans). We still need probability to discern “the odds” of that event (e.g. dirty nuclear bomb in Los Angeles), so Bayes is helpful because we can infer subjective probabilities or “the possible value of unknowns” from similar situations tangential to our own predicament. Lastly, there are “unknown unknowns”, or things we haven’t even dreamed about (Taleb’s Black Swan). Dr. Nassim Nicholas Taleb labels this “the fourth quadrant” where probability theory has no answers. What’s an illustration of an “unknown unknown”? Dr. Taleb gives us an example of the invention of the wheel, because no one had even though or dreamed of a wheel until it was actually invented. The “unknown unknown” is unpredictable, because—like the wheel—had it been conceived by someone, it would have been already invented. Rumsfeld’s quote gives business managers a framework for thinking probabilistically. There are “known knowns” for which Frequentism works best, “unknown knowns” for which Bayesian Inference is the best fit, and there is a realm of “unknown unknowns” where statistics falls short, where there can be no predictions. This area outside the boundary of statistics is the most dangerous area, says Dr. Taleb, because extreme events in this sector usually carry large impacts . . .

I add, that's where you need creative strategic thinkers who can suss out subtle signs and bring them to the table. Onward, I would lay out a game tableau and/or decision tree for a multi-player, multi-turn game against known players AND "nature" involving not only known patterns but potential high impact outliers brought to the table by the futurists. Then, we can proceed to a scenario based analysis on the outcome patterns of business as usual vs credible alternatives including black swans as well as internal-to-dynamics catastrophes. All of this is cast in the context of strategic decision-making. How we apply this to science is to imagine ourselves sitting on the board of a big-ticket journal, and deciding on what to publish; where, your journal also has a research grant budget so it can help shape the path of research. (High status institutions in Science do indirectly influence research funding.) Then, factor in the prudential principle of least regret on the downside and biggest windfall on the upside. Now, recommend your bets on an imaginary pool of grant funds for research and your recommendations on what to publish, knowing that this is likely to shift the balance of grant funding for future turns. What articles and proposals would you entertain, why? What SHOULD you be doing to hedge, to minimise likelihood of big regrets on both the downside and the up-side? Now, imagine you are the interested public, voting for the pols who provide the Journal's grant pool and influence priorities in part influenced by the journal's publications. Which pols will you most likely vote for, why? Which SHOULD you vote for, why? See the emergence of a problematique of thorny interacting, mutually reinforcing problems that can easily, blindly reinforce policies and ideologies that amplify exposure to destructive black swans? Now, look back at the OP and particularly the appended charts. Tell us if a lightbulb goes off. KF kairosfocus

MR, you were repeatedly directed to Taleb and the black swan concept. If in a chaotic age you do not have an inkling about radical uncertainty with high potential impact and the need to respond to adequate warrant for the unwelcome, you need to come in out of the rain. KF kairosfocus

I'm afraid I can't extract any signal from this noise, so I guess I'll leave you to it. mullers_ratchet

JS, it should be clear that there are limits to statistical, probabilistic approaches such as Bayesian and likelihood approaches in general;which were raised by MR in response to the epistemological corrective highlighted in the OP. If you are not familiar with Taleb's Black Swan book and linked discussions, I suggest that you may find it useful to do some reading. Though in fact my own knowledge base on this comes from issues in strategic planning, finance, macroeconomics and sustainable development, also philosophy. All, i/l/o events on July 18, 1995 and thereafter, when a discontinuity in history utterly and radically reshaped the history of this nation, utterly invalidating all confident projections that were made hitherto -- and exposing the folly of business as usual and linked dismissiveness in the years that followed, ultimately costing people their lives. Well do I recall hearing radio discussions by ministers of government on how those entertaining visions of catastrophe were exhibiting neurotic behaviour and even notifying the public of plans for "subversives" -- the people who were trying to warn that we were dealing with a highly dangerous unknown and that optimistic probability estimates were ill-founded. The evidentialist fallacy has serious consequences. KF kairosfocus

MR, with all due respect what I cited from Taleb and summarised from my own background is highly relevant to the "extraordinary claims" assertion by Sagan et al, and to the putting up of Bayesian analysis as a scientific cover for ideologically driven selective hyperskepticism. The point is, again, that the cases where we can construct a plausible probability distribution and use probability revision techniques or likelihood analysis are the nicely-behaved cases. Far too often we have to deal with uncertainty which opens up Taleb's Black Swans and Boyd's broken OODA loop. Rare, high impact, unexpected events are often an utterly dominant fact of life that shatters extrapolations and nicely calculated models. In that light, it is utterly imprudent to disregard adequate warrant for a result because the conclusion it points to is unwelcome and perceived as "extraordinary" leading to ill-advised imposition of arbitrarily biased hyperskepticism and usually groupthink to back it up. This obtains for ordinary life, for actuarial mathematics, for the court room, for strategic management, for sustainable development planning, for finance and investment, for programmes and projects, for macroeconomic planning, for government generally, for national security and the military, for engineering and for science. KF kairosfocus

Well, JSmith, claiming FTL travel is an extraordinary claim outside of science fiction. And demonstrating FTL travel would be extraordinary given our current understanding. Claiming blind and mindless processes produced the variety of life starting from some simple replicators is an extraordinary claim. And given what we know about living organisms demonstrating such a thing would indeed be extraordinary. ET

KF at 44 and 45, maybe I am clueless, but what does any of this have to do with whether or not extraordinary claims require extraordinary evidence. Too many of these threads get dragged off on bizarre tangents. JSmith

Are you OK? This is not related to the topic of discussion at all, as far as I can tell. mullers_ratchet

PPS: More from Taleb:

THE FOURTH QUADRANT: A MAP OF THE LIMITS OF STATISTICS (2008) Statistical and applied probabilistic knowledge is the core of knowledge; statistics is what tells you if something is true, false, or merely anecdotal; it is the "logic of science"; it is the instrument of risk-taking; it is the applied tools of epistemology; you can't be a modern intellectual and not think probabilistically—but... let's not be suckers. The problem is much more complicated than it seems to the casual, mechanistic user who picked it up in graduate school. Statistics can fool you. In fact it is fooling your government right now. It can even bankrupt the system (let's face it: use of probabilistic methods for the estimation of risks did just blow up the banking system). The current subprime crisis has been doing wonders for the reception of any ideas about probability-driven claims in science, particularly in social science, economics, and "econometrics" (quantitative economics). Clearly, with current International Monetary Fund estimates of the costs of the 2007-2008 subprime crisis, the banking system seems to have lost more on risk taking (from the failures of quantitative risk management) than every penny banks ever earned taking risks. But it was easy to see from the past that the pilot did not have the qualifications to fly the plane and was using the wrong navigation tools: The same happened in 1983 with money center banks losing cumulatively every penny ever made, and in 1991-1992 when the Savings and Loans industry became history. It appears that financial institutions earn money on transactions (say fees on your mother-in-law's checking account) and lose everything taking risks they don't understand. I want this to stop, and stop now . . .

--> BTW, we may be going through a Black Swan right now; with the discovery of a major architecture flaw in many families of microprocessors. kairosfocus

MR, on Sept 10, 2001, what was the predicted likelihood of what happened the next day and changed the course of history? November 21, 1963? December 6, 1941, June 27, 1914? Just to pick one class of explosive break-points. (cf. here: https://www.quora.com/What-are-some-of-the-most-important-black-swan-events-in-history ) I am pointing out that likelihood ratios or Bayesian probability revisions or other approaches do not map uncertainty as opposed to risk very well, and that the Black Swan concept is a very important counter-weight to the notion that we have a handle on things down to a probability distribution. You raised issues on this line, I have responded and pointed to why although expert elicitation and the like can be useful, scenario based examination of possibilities under radical uncertainty and the capacity to be critically aware, intuitively open minded, deeply insightful and more built up through such are even more important. As in, John Boyd has a very serious set of points. And yes I am looking far beyond nice neat comparative hyps and likelihood ratios etc. That is to show in outline that Bayesian approaches have problems and beyond that this is actually a side-track from the demonstrable agenda of selective hyperskepticism in pursuit of ideological imposition. Or have you forgotten who wrote the book, The Demon-Haunted World? Where also, FYI, troll tolerance has gone to zero, if you think that clever insubstantial dismissive talking points are going to cut it. If you have substantial concerns, raise and address them. If not, don't take steps that will put you in the same box with the animus-driven trolls. KF PS: Taleb:

What we call here a Black Swan (and capitalize it) is an event with the following three attributes. First, it is an outlier, as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. [ --> see the problem of ideologically loaded question-begging evidentialism that makes a crooked yardstick the standard of straightness and accuracy, in the teeth of adequate evidence?] Second, it carries an extreme 'impact'. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable. I stop and summarize the triplet: rarity, extreme 'impact', and retrospective (though not prospective) predictability. A small number of Black Swans explains almost everything in our world, from the success of ideas and religions, to the dynamics of historical events, to elements of our own personal lives. [The Black Swan, ch 1. Notice, ability to actually spot and use subtle signs would be enormously relevant, but the problem is we characteristically don't spot them or take them seriously. On Dec 7, 1941, the IJN raid was spotted on radar and it should have been known that radar had saved Britain the previous year, but the watch supervisor insisted it had to be an expected flight of B17's. I trust this helps you see why I corrected Sagan as in the OP.]

kairosfocus

Yes, duller, you are sorry. And just because you are too dull to understand the answer doesn't mean it hasn't been given. In comment 37 you talked about evidence that was never previously provided. So if you want to eat word soup start with yours. ET

I'm sorry, this is a word soup that doesn't answer the question at all. mullers_ratchet

PS: Cf OP addendum on U-UBSE and envisioning uncertainty. kairosfocus

MR, imposing a Bayes Theorem/Likelihood grid may be applicable to some situations but in fact the issues originally in view and in view as applied today are much broader. One key facet is the fuzzy border between risk and uncertainty. Risk deals with unknowns sufficiently understood to have some sort of reasonable probability estimation. Uncertainty deals with unknowns that we may be aware of but which are intractable and worse yet the unknown unknowns that we cannot see for any one of many reasons. In short the Bayesian/Likelihood model you have tried to impose to make Sagan's version of Cliffordian evidentialism seem more plausible and thus apparently reasonable and responsible -- when manifestly it is often used to impose ideologically driven hyperskeptical blinkers and soft nihilist power games -- simply does not fit the relevant domain. KF kairosfocus

MR @ 37: I could see one thinking that a claim "out of the ordinary" would require evidence that "violates the ordinary", with an expectation that "the ordinary" is well understood and explained. However, it isn't. Simple example: The precession of Mercury. A readily observable phenomenon for centuries; explained often, maybe even to the satisfaction of many, never actually understood. Then comes along the extraordinary claim of General Relativity; a very counterintuitive little piece of mathemagic; to explain this and some other rather "ordinary" evidences. LocalMinimum

duller:

Given those two pieces of evidence, do you think both claims are equally likely?

What evidence? ET

I guess I'll ask you the same question as I did Joe in 34. Given those two pieces of evidence, do you think both claims are equally likely? Also, do you accept the "extraordinary evidence" phrase describes the application of Bayes Theorem (and that Bayes Theorem is true). mullers_ratchet

MR, kindly stop trying to muddy a clear issue. It is a well known standard of empirical science that experimental results should be confirmed by multiple parties, preferably using diverse techniques. That criterion was never met. Observational sciences cannot repeat experiments so have weaker warrant. And, reconstructions of the remote, unobserved past of origins have yet weaker warrant. This is of course not far from a summary of Newton's rules. Of course, this has been too often suppressed in recent times and those who point it out have been unjustly pounced on in the media etc and treated as pariahs. KF kairosfocus

muller- You have issues. You just spewed nonsense about what kairosfocus posted. Just own up to that fact. For the record, I tend not to believe anyone. Trust has to be earned. Someone tells me they flipped a coin and it came up heads I would ask them if that meant it was time to change their undies. Someone tells me they ran an experiment and found FTL travel and I would ask to see the experiment run again and again. ET

OK Joe. Someone tells you they tossed a coin and it came up heads. Someone else tells you they ran an experiment that showed faster than light travel is possible. Do you believe each claim equally? mullers_ratchet

muller:

So… because the evidence wasn’t very strong and the prior probability was so low you dismissed the result.

That doesn't follow from what you quoted. Do you have moral issues? ET

I should point out, the "extraordinary evidence" phrase is just an informal way of describing an application of Bayes Theorem. I think even ID is probably not going to deny elementary mathematical theorems! mullers_ratchet

DS, the original reports were tentative, and were soon withdrawn. There was no serious reason for a follow-up. If there had been confident reports and confirmation, that would be opening up a different context. KF

So... because the evidence wasn't very strong and the prior probability was so low you dismissed the result. But if the evidence was stronger (more "confident" or "confirmed") you would be more open to changing your view. Almost as if a priori improbable claims require stronger evidence than others... mullers_ratchet

Dick:

Suppose someone were to assert that Sagan’s claim (actually Hume’s) that extraordinary claims require extraordinary evidence is itself an extraordinary claim.

Then they would have to support it. If Sagan/ Hume didn't present anything to support their claim then it can be ignored. ET

Suppose someone were to assert that Sagan's claim (actually Hume's) that extraordinary claims require extraordinary evidence is itself an extraordinary claim. Would they be wrong, and, if not, what's the extraordinary evidence that supports Sagan's claim? Dick

DS, the original reports were tentative, and were soon withdrawn. There was no serious reason for a follow-up. If there had been confident reports and confirmation, that would be opening up a different context. KF kairosfocus

UB, yup. KF kairosfocus

some sort of (e)mbedded mathematics, geometry or language certainly would be adequate.

Language requires a very specific physical organization. This organization is not only physically identifiable, but is exclusively found only in language systems -- with one exception. It is also the physical organization that enables the gene system. So, the very thing that would make a SETI claim of 'intelligent action' irrefutable, is found at the very origin of the living cell. And the double standards suddenly appear. Upright BiPed

Where you stand depends on where you sit when it comes to applying the adjective, I suppose. KF's point that Sagan intended it to rationalize hyperskepticism is spot on to anyone paying attention. That said, personally, I find the design in biology and the fine tuning of the cosmos that have been uncovered since Sagan's demise to be extraordinary evidence for God. So strong, that his materialist heirs have been forced to extraordinary lengths to argue against it. A multiverse hypothesis that will forever remain impossible to verify experimentally? Extraordinary! jstanley01

KF, PS to my #19, It's also clear that everyone, including you and me, quickly accepted the explanation that the original experiment was flawed, and that the results were actually consistent with GR. To summarize, if the headline states "Experiment possibly falsifies GR!", then we are very cautious and wait for further confirmation. If the headline is "Experiment produces results consistent with GR!", then we are more likely to accept it at face value. daveS

OK got it- FTL should have given it away ET

withstood 100+ years of tests

And what do you mean by "withstood"? What "tests"? If you mean the goalposts continue to be moved, I might agree. Andrew asauber

ET, General relativity. daveS

daveS:

I think most of us in practice would be especially cautious in the treatment of evidence which would overturn a theory that has withstood 100+ years of tests (and rightly so).

And what theory would that be? ET

KF,

There is no more duty of care to be meticulous and correct on “extraordinary” results than on “ordinary” ones; warrant cannot accept double standards on evidence. Both are governed by the same duties of care of sound work towards truth.

I think most of us in practice would be especially cautious in the treatment of evidence which would overturn a theory that has withstood nearly 100 years of tests (and rightly so). daveS

And Darwin's idea still lacks evidentiary support. The dinosaur killing asteroid has also been changed to say that most dinos died before the impact as evidenced by the total lack of dino fossils in and just above the KT boundary. ET

KF

DS, There is no more duty of care to be meticulous and correct on “extraordinary” results than on “ordinary” ones; warrant cannot accept double standards on evidence. Both are governed by the same duties of care of sound work towards truth.

Extraordinary is just an adjective qualifying the degree to which a claim differs from currently "accepted" knowledge. When heliocentrism was first proposed, it was an extraordinary claim. It required extraordinary evidence to become accepted. Plate tectonics was the same. As was the dinosaur killing asteroid and Darwin's theory. If we didn't require extraordinary evidence in order to accept an extraordinary claim, one that constitutes a fundamental change in our models, science would never advance. Admittedly, this will often delay the acceptance of a more accurate model, but it will never stop the process. JSmith

JSmith:

The same applies to ID. I think that we would all agree that life arising due to intelligent design would be an extraordinary claim.

And ID has the methodology to test that claim, it has been tested and living organisms were intentionally designed.

As is the claim that it arose naturally.

There isn't even a methodology to test such a claim. As for SETI it too has a methodology. Looks like only materialism lacks one. ET

JS, the evidence is clear: the "extraordinary evidence" standard has been used to insert a double-standard of warrant, leading to unwarranted exclusion of unwelcome results. What would be reasonable is to exert care based on potential consequences of error, i.e. moral certainty is the relevant standard when the cost of error is too great or would be unjust. Do not forget, I bear the name of a man judicially murdered on one hour's notice to himself when he was blocked from accessing the testimony of his physician; whose report on the nephritis he was suffering would have utterly devastated the case being made against him in the military court. KF kairosfocus

DS, There is no more duty of care to be meticulous and correct on "extraordinary" results than on "ordinary" ones; warrant cannot accept double standards on evidence. Both are governed by the same duties of care of sound work towards truth. KF kairosfocus

KF,

DS, the standard degree of credible empirical warrant. There was a recent case and there turned out to be an error. KF

If it was your work, would you not have scrutinized the FTL experiment with extraordinary care, compared to one which supported a less surprising conclusion? daveS

JG

Interesting – which kind of evidence is (in principle) more extraordinary: that one should pick up a signal originating from an intelligent lifeform somewhere in a universe the size of ours, or that one should not pick up such a signal?

I wasn't intending to drag this into a SETI thread. If you prefer, I will give another example. A claim that Bigfoot is real and living in Montserrat would definitely be an extraordinary claim. A blurry picture with no landmarks would not be adequate to compellingly support the claim .Given current digital manipulations, a clear picture with identifiable landmarks would also not be adequate. However, this, along with tissue samples from the same location that have DNA that cannot be linked to any other species of animal alive would be extraordinary, and adequate as compelling evidence. The same applies to ID. I think that we would all agree that life arising due to intelligent design would be an extraordinary claim. As is the claim that it arose naturally. At present, there is no evidence to compellingly support one claim over the other. I think that we would agree that for any evidence to be compelling enough to discard one or the other as possibilities would have to be extraordinary evidence. I just don't see any problem with the way that Sagan phrased his statement. JSmith

Interesting - which kind of evidence is (in principle) more extraordinary: that one should pick up a signal originating from an intelligent lifeform somewhere in a universe the size of ours, or that one should not pick up such a signal? Isn't the whole astrobiology project based on the assumption that life ought to be as common as muck out there? Jon Garvey

We are just talking semantics here. Any evidence compelling enough to be adequate to support an extraordinary claim would, itself be extraordinary. For example, A claim that SETI had irrefutably identified an intelligent signal from space would definitely be an extraordinary claim. Non-randomness in the signal would not be adequate evidence to make this claim. Nor would non-randomness and repeated sequences. However, some sort of imbedded mathematics, geometry or language certainly would be adequate. And that would also be extraordinary evidence. JSmith

Interesting that Sagan had no trouble believing in abiogenesis theory despite there being zero evidence for it. He certainly didn't demand extraordinary evidence for that faith-based idea. Truth Will Set You Free

DS, the standard degree of credible empirical warrant. There was a recent case and there turned out to be an error. KF kairosfocus

KF, While I agree with the OP to some extent, I also think different people are using the terms "extraordinary claims|evidence" in different ways. Suppose I claimed to be able to send signals from Milan to Minsk at 10 times the speed of light in a vacuum. Is that an extraordinary claim? What sort of evidence would you consider adequate to demonstrate this claim? daveS

DS, there are direct indications in the TED video -- right in the opening -- that the above is still needed. KF kairosfocus

nm--already answered. daveS

Good post. Sagan was simply revealing his extraordinary a/mat bias. Truth Will Set You Free

Dick, that goes to the heart of the matter. What seems extraordinary is going to be what is not expected on the dominant perspective or school of thought at work. Sagan's declaration is little more than a policy declaration that selective hyperskepticism will be used to lock out what is unwelcome, by exerting an arbitrarily and inconsistently high standard of warrant for what one is inclined to reject. Too often, by recognising the sort of warrant that will be feasible for that sort of claim then pushing the bar far higher, studiously evading that if such were consistently applied, the relevant field of thought or action would collapse. Cases in point, sadly, are legion. KF PS: And when one inappropriately rejects what on reasonable and feasible warrant one should believe, of course one is led to cling to what one should not believe. A double-whammy. kairosfocus

What constitutes "extraordinary" evidence anyway? What are the criteria which establish evidence as "extraordinary"? Dick

Tabby’s Star — on the “extraordinary evidence” claim kairosfocus