Climate change Neuroscience

How much attention should we pay to pundit predictions?

Spread the love

Maybe not so much. Jonah Lehrer, contributing editor at Wired, and blogger at The Frontal Cortex writes,

In the early 1980s, Philip Tetlock at UC Berkeley picked two hundred and eighty-four people who made their living “commenting or offering advice on political and economic trends” and began asking them to make predictions about future events. He had a long list of pertinent questions. Would George Bush be re-elected? Would there be a peaceful end to apartheid in South Africa? Would Quebec secede from Canada? Would the dot-com bubble burst? In each case, the pundits were asked to rate the probability of several possible outcomes. Tetlock then interrogated the pundits about their thought process, so that he could better understand how they made up their minds. By the end of the study, Tetlock had quantified 82,361 different predictions.

After Tetlock tallied up the data, the predictive failures of the pundits became obvious. Although they were paid for their keen insights into world affairs, they tended to perform worse than random chance. Most of Tetlock’s questions had three possible answers; the pundits, on average, selected the right answer less than 33 percent of the time. In other words, a dart-throwing chimp would have beaten the vast majority of professionals. Tetlock also found that the most famous pundits in Tetlock’s study tended to be the least accurate, consistently churning out overblown and overconfident forecasts. Eminence was a handicap.

Lehrer worries that bad expert advice can “reliably tamp down activity in brain regions” that monitor errors and mistakes.

Rest here.

For Tetlock, go here.

I am skeptical of the mechanistic, brain-based explanation Lehrer offers. People often believe things because the social rewards of belief are greater than the social rewards of disbelief.

For example, if I said that I didn’t believe that polar bear numbers are drastically decreasing (see also here), some people out there would assume that I enjoy torturing kittens on my break, and would not accept my view as a considered judgement. And if they can find a pundit to back them up, that is all they need. The problem is that they then vote for public policy that might not work out the way they hope.

Here is an example:

The United States now bans the import of polar bear skins, to protect the bear. I am sure many New York socialites toasted the decision. Maybe prematurely. The former practice of Inuit (Canadian Eskimos) was to sell their bear tag (the ancestral right to kill a bear) to a wealthy American hunter, an important source of income for communities much poorer than the socialite’s.

The American, much of the time, never shoots a bear and just goes home. Then the tag is forfeit. So the number of tags in circulation is not equal to the number of dead bears, and is a poor predictor of population size. However, with no American market, the local hunter keeps his tag – and keeps on trying until he shoots a bear. For more on this and related issues around counting bears, see this Maclean’s article. (Maclean’s is our national news magazine.)

I suppose it is some consolation that at least the number of tags in circulation will now more closely correlate to known bear deaths. Of course, more bears will die, but that is a small price to pay for the opportunity to be guided by the pundit.

Hat tip: Stephanie West Allen at Brains on Purpose

12 Replies to “How much attention should we pay to pundit predictions?

  1. 1
    Collin says:

    Often pundits and some critics of ID (and probably some critics of evolution) use the following logical fallacy:

    poisoning the wells — from an article by Albury Castell titled “Analyzing A Fallacy,” which was included in the book Readings In Speech, edited by Haig Bosmajian (Harper & Row, 1965). Here is the full quote: “During the last century a famous controversy took place between Charles Kingsley and Cardinal Newman. It began, I believe, by Kingsley suggesting that truth did not possess the highest value for a Roman Catholic priest; that some things were prized above truth. Newman protested that such a remark made it impossible for an opponent to state his case. How could Newman prove to Kingsley that he did have more regard for truth than for anything else, if Kingsley argued from the premiss that he did not? It is not merely a question of two persons entertaining contradictory opinions. It is subtler than that. To put it baldly, Newman would be logically ‘hamstrung.’ Any argument he might use to prove that he did entertain a high regard for truth was automatically ruled out by Kingsley’s hypothesis that he did not. Newman coined the expression poisoning the wells for such unfair tactics…The phrase poisoning the wells exactly hits off the difficulty. If the well is poisoned, no water drawn from it can be used. If a case is so stated that contrary evidence is automatically precluded, no arguments against it can be used.”

  2. 2
    Retroman says:

    Denise, why do you doubt his brain-based neuroscience? It seems to me that he has pretty good evidence for what he says. I just ask, because I like to be guided by evidence, generally.

  3. 3
    O'Leary says:

    Retroman at 2, I have pretty good evidence for what I say too.

    Anyone does, who doubts punditry, however they arrivea t the discovery that punditry does not amount to successful prediction.

    People believe pundits for a variety of reasons; I know better than to think there is a single signature in the brain for that sort of thing.

    So far as I can tell, Lehrer is a professional writer, like me, and entitled to his opinion.

  4. 4
    Retroman says:

    Sure, but he gives hard evidence to back up his opinion. What evidence do you have for what you say beyond hypotheticals?

  5. 5
    Clive Hayden says:

    Retroman,

    Denise, why do you doubt his brain-based neuroscience? It seems to me that he has pretty good evidence for what he says. I just ask, because I like to be guided by evidence, generally.

    What evidence do you cite that tells you that you should be guided by evidence? The personal subjective notion that you just “like” to be guided by evidence? But of course “like” is not evidence itself. My point is that, we cannot take evidence for everything that we think, for then there would exist no thinking on its own to even begin or continue the process objectively. All thoughts would be subject to the same thing, and none of it would be able to be an objective reference, to “stand outside” and make judgments, even on evidence. You cannot control the river if you’re a single drop of water, and the same would go for materialistic notions of thoughts.

  6. 6
    Retroman says:

    Clive, I am guided by evidence for a simple reason: experience and practicality has shown that answers that are guided by evidence are correct far more often.

    But a question for you: are you suggesting that I am wrong to prefer evidence?

  7. 7
    vjtorley says:

    Retroman:

    I had a look at the online article, Given “Expert” Advice, Brains Shut Down , which Lehrer cites as evidence for his brain-based model of faulty decision-making. The nice thing about it is that at the end it proposes an experiment that could falsify the materialist acocunt.

    In the study, Berns’ team hooked 24 college students to brain scanners as they contemplated swapping a guaranteed payment for a chance at a higher lottery payout. Sometimes the students made the decision on their own. At other times they received written advice from Charles
    Noussair, an Emory University economist who advises the U.S. Federal Reserve…

    When thinking for themselves, students showed activity in their anterior cingulate cortex and dorsolateral prefrontal cortex — brain regions associated with making decisions and calculating probabilities. When given advice from Noussair, activity in those regions flat lined…

    In the future, however, Berns said it might be possible to construct a brain-scanning device that lets people know when their decision-making faculties go to sleep.

    The question then, he said, is whether people would still make sensible decisions.

    “That’s a great study,” said Berns. “Maybe I’ll apply for some stimulus money to do it.”

    Care to volunteer?

    If an immaterialist account of human thought is correct, people could train themselves to resist changes that they knew were happening in their brains. That would lend support to a top-down, immaterialist account of the mind, though it would not be conclusive in and of itself.

    By the way, it may interest you to know that Denyse has co-written a book with Canadian neuroscientist Mario Beauregard, entitled The Spiritual Brain . The book documents instances of how people can sometimes change their brain chemistry simply by changing their attitudes. So I think Denyse’s skepticism of simplistic, bottom-up, materialistic explanations for faulty decision-making is entirely justifiable.

  8. 8
    Clive Hayden says:

    Retroman,

    But a question for you: are you suggesting that I am wrong to prefer evidence?

    I am saying that value judgments, words like “should” have no equivalent evidence. You have to start without evidence, indeed with a philosophy, in the first place, before you can make the philosophical assertion that evidence has any value, or that value has any value for that matter. But none of this is itself “evidence” in the material world.

  9. 9
    Retroman says:

    vjtorley, thank you. I didn’t know about Denyse’s book. I wish she’d have pointed it out to me, instead of just giving hypotheticals. I will see if my local library points it out.

    Clive, one doesn’t need to start with any philosophy to see what works and what doesn’t. Well, I suppose one has to start with the idea that there is an outside world and that sollipsism is false, but other than that, no. Still, you value statements backed up by evidence more than ones that are unsupported by good evidence, don’t you? I get the impression that you are disagreeing simply to disagree, or to push an agenda. I’m not sure why you’re doing that.

  10. 10
    Retroman says:

    Errr, I mean “I’ll see if my local library has it.” I need coffee. : )

  11. 11
    Clive Hayden says:

    Retroman,

    Clive, one doesn’t need to start with any philosophy to see what works and what doesn’t. Well, I suppose one has to start with the idea that there is an outside world and that sollipsism is false, but other than that, no. Still, you value statements backed up by evidence more than ones that are unsupported by good evidence, don’t you?

    Yes, you start with a philosophy. You can’t have it any other way. The philosophy is one that tells you that you “should” even begin to attempt to “find something out”, and then that you “should” use something like “evidence,” and that there is value in both of those philosophical propositions by comparison to another contrary methodology. Science cannot justify why you should do science.

  12. 12
    Retroman says:

    Well my biology tells me to eat, to avoid pain, to seek comfort, and so forth. It’s built into me and provides all the non-philosophical motivation I need to seek ways to provide things for myself. Science has proven to be a pretty good way to provide these things.

    But I keep asking you, and you keep ignoring the question: you value statements backed up by evidence more than ones that are unsupported by good evidence, don’t you?

    Your philosophical tangent was started as a request for Denyse’s evidence. I am not quite sure why you started this tangent (except maybe to cast some sort of dispersion on science or on asking for evidence before believing something– I’m not sure). Anyway, care to answer my first question, then tell me what the point of your tangent was?

Leave a Reply