Eric MacDonald, a former Church of England clergyman who is now an atheist, knows that prayer doesn’t make sick people better. Dr. John Polkinghorne, KBE, FRS, a former physics professor at Cambridge who is also an Anglican priest and theologian, who recently wrote Questions of Truth: Responses to Questions about God, Science and Belief (Westminster John Knox Press, 2009), in collaboration with Nicholas Beale, FRSA, believes that at times, prayer “may” result in “remarkable physical recovery,” although often it does not (see the “Science of Prayer” video here). Whom should we believe?
In yesterday’s post, which was entitled, Is this the Dumbest Ever “Refutation” of the Fine-Tuning Argument?, I exposed the silly fallacies in atheist philosopher Anthony Grayling’s criticisms of the cosmological fine-tuning argument, which Dr. Polkinghorne defends in his book. Today, I’ll be focusing my sights on Eric MacDonald’s critical review of Polkinghorne’s book.
In a provocatively titled post, Polkinghorne: Religion, Lies and Digital Video (17 October 2011), MacDonald accuses Dr. Polkinghorne of lying to his audience:
Praying — as Polkinghorne must know — has no effect whatever on disease outcomes, and if it’s just a kind of psychological self-help program he should say so, instead of resorting to the mumbo-jumbo of religion….
… Polkinghorne must know, if anything can be called wish-fulfilment, religion is a prime candidate, since there isn’t a shred of evidence to show that religion is true….
… Polkinghorne is deliberately misleading his audience. In any other language this is called lying, and to my mind, it just shows how desperate religious believers are, that they can shamelessly tout their beliefs in this hucksterish way, like a carnie in the midway of a country fair.
“Lying.” That’s a pretty strong accusation to make against an Anglican priest who is also a knight and a Fellow of the Royal Society. In my language, that would be called slander. It might help MacDonald’s case, though, if his charges against Dr. Polkinghorne turned out to be true. But are they?
Watching the video, I was reminded of a statement made by the late Arthur C. Clarke:
When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
(Clarke’s first law, which he originally proposed in the essay, “Hazards of Prophecy: The Failure of Imagination”, in Profiles of the Future, 1962.)
Dr. Polkinghorne is 81. Should we believe him when he says that prayer may result in a remarkable physical recovery?
In order to answer that question, it might help to do something that MacDonald forgot to do: look at the evidence. The Wikipedia article, Efficacy of prayer turned out to contain a couple of surprising nuggets of information:
Determining the efficacy of prayer has been attempted in various studies since Francis Galton first addressed it in 1872. Some studies have reported benefit, some have reported harm, and some have found no benefit from the act of praying. Others suggest that the topic is outside the realm of science altogether…
In comparison to other fields that have been scientifically studied, carefully monitored studies of prayer are relatively few. The field remains tiny, with about $5 million spent worldwide on such research. If and when more studies of prayer are done, the issue of prayer’s efficacy may be further clarified.
In the absence of decisive results one way or the other, a prudent agnosticism would surely be the healthiest scientific attitude to adopt. When Eric MacDonald claims to “know” that praying “has no effect whatever on disease outcomes,” he is not speaking in the way that a scientist would talk.
Atheists who write on the subject of prayer commonly cite the 2006 Benson study as the last word on the subject of intercessory prayer. This study found that intercessory prayer had no effect on recovery from surgery without complications, and that patients who knew they were receiving
intercessory prayer actually fared worse. (Benson H, Dusek JA, Sherwood JB, et al. (April 2006). “Study of the Therapeutic Effects of Intercessory Prayer (STEP) in cardiac bypass patients: a multicenter randomized trial of uncertainty and certainty of receiving intercessory prayer”. American Heart Journal 151 (4): 934–42. doi:10.1016/j.ahj.2005.05.028. Abstract. Lay summary.) One curious feature of this study, unlike previous studies, was that the intercessors were not allowed to pray their own prayers. The prayers were given to them by the study coordinators, in order to “standardize” the prayers. The discussion section of the study acknowledged that some of the intercessors were dissatisfied with the canned nature of the prayers.
In 2007, a systemic review of 17 studies (including Benson’s) stated that there are “small, but significant, effect sizes” for the use of intercessory prayer in the reviewed literature. (David R. Hodge, “A Systematic Review of the Empirical Literature on Intercessory Prayer” in Research on Social Work Practice, March 2007, vol. 17 no. 2, 174-187 doi: 10.1177/1049731506296170.) For the benefit of readers, I’ll quote a few of the study’s highlights:
Inclusion and Exclusion Criteria
Because the purpose of the review was to examine research capable of informing and guiding practice decisions, studies had to meet the following criteria to be included in the review: (a) use intercessory prayer as an intervention, (b) implement the intervention with a population of clients or patients, and (c) test the efficacy of the intervention, preferably using standardized measures and a double-blind randomized control trial (RCT) methodology. (p. 175)
Findings Supportive of Prayer
Individual assessment revealed that patients who received intercessory prayer demonstrated significant improvement compared to those who received standard treatment devoid of prayer in 7 of the 17 studies. Furthermore, in an additional 5 studies, the trend favored the prayer group. This raises the possibility that an increase in power would yield significant findings. (p. 182)
Findings Unsupportive of Prayer
Conversely, in 10 of the studies, prayer was unassociated with positive improvement in the condition of clients. In addition, in many of the studies in which significant results were obtained, the results were not uniformly positive across outcome variables. For instance, in the Byrd (1988) study, only six positive outcomes were recorded among 26 specific problem conditions. This type of inconsistent pattern raises the possibility of Type I errors. (p. 182)
The Use of Informed Consent for Private Intercessory Prayer
With the exception of one small pilot study (i.e., Mathai & Bourne, 2004), all six studies in which clients were completely unaware of the intervention yielded positive outcomes or exhibited a trend in favor of the group receiving intercessory prayer. This finding held irrespective of when the prayer was offered (prospective vs. retrospective) or the spiritual tradition of those providing the prayer (Quakers vs. Catholics). (p. 184)
Conclusion
Intercessory prayer offered on behalf of clients in clinical settings is a controversial practice, in spite of its apparent frequent occurrence. The topic is one that engenders both support and opposition, often passionately held. This study has attempted to shed some light on the controversy by examining the empirical literature on intercessory prayer.
Practitioners who adhere to Division 12 criteria have little basis for using intercessory prayer, in spite of a meta-analysis indicating small, but significant, effect sizes for the use of intercessory prayer. Most practitioners, however, are likely to affirm the broader understanding of evidence-based practice articulated in the APA’s Presidential Task Force on Evidence-based Practice (2006). Such practitioners may believe that the best available evidence currently supports the use of intercessory prayer as an intervention.
Thus, at this junction in time, the results might be considered inconclusive. Indeed, perhaps the most certain result stemming from this study is the following: The findings are unlikely to satisfy either proponents or opponents of intercessory prayer.
Faced with evidence of this nature, I submit that an honest scientist would withhold judgement regarding the efficacy of prayer. When Dr. Polkinghorne modestly asserts his belief that at times, prayer “may” result in “remarkable physical recovery,” he is perfectly within his rights as a scientist.
Before I conclude my discussion of prayer studies, I’d like to mention two complicating factors which make the interpretation of prayer studies difficult. The first is that even in a control group, family members will almost certainly be praying for the patients, if the study is being conducted in North America. Perhaps it would be better to conduct a study in China, whose government is officially atheist, if one wants better controls.
The second complicating factor underlying prayer studies is the questionable assumption that a Deity (supposing Him to be the One responsible for answering prayers) would be more inclined to help those who pray than those who do not. Even Holy Scripture is not clear about this: we are told that “The prayer of a righteous person is powerful and effective” (James 5:16, NIV), but we are also told that God “causes his sun to rise on the evil and the good, and sends rain on the righteous and the unrighteous” (Matthew 5:45). Prayers may indeed be powerful, but whom do they help? God alone knows.
The Power of Prayer – the Experience of C. S. Lewis
One person who had quite a lot to say on the power of prayer was C. S. Lewis, a former atheist who was highly doubtful of the worth of intercessory prayer studies, but who had personal experiences which led him to affirm that prayer actually works. Lewis’ description of his experiences in his essay, The Efficacy of Prayer, with prayer makes for intriguing reading:
Some years ago I got up one morning intending to have my hair cut in preparation for a visit to London, and the first letter I opened made it clear I need not go to London. So I decided to put the haircut off too. But then there began the most unaccountable little nagging in my mind, almost like a voice saying, “Get it cut all the same. Go and get it cut.” In the end I could stand it no longer. I went. Now my barber at that time was a fellow Christian and a man of many troubles whom my brother and I had sometimes been able to help. The moment I opened his shop door he said, “Oh, I was praying you might come today.” And in fact if I had come a day or so later I should have been of no use to him.
It awed me; it awes me still. But of course one cannot rigorously prove a causal connection between the barber’s prayers and my visit. It might be telepathy. It might be accident.
I have stood by the bedside of a woman whose thighbone was eaten through with cancer and who had thriving colonies of the disease in many other bones, as well. It took three people to move her in bed. The doctors predicted a few months of life; the nurses (who often know better), a few weeks. A good man laid his hands on her and prayed. A year later the patient was walking (uphill, too, through rough woodland) and the man who took the last X-ray photos was saying, “These bones are as solid as rock. It’s miraculous.”
But once again there is no rigorous proof. Medicine, as all true doctors admit, is not an exact science. We need not invoke the supernatural to explain the falsification of its prophecies. You need not, unless you choose, believe in a causal connection between the prayers and the recovery….
Cases like these are singular, but they certainly give one pause. Whatever one makes of them, they shatter the shrill dogmatism of ex-clergymen like MacDonald, who claim to know that prayer doesn’t affect disease outcomes.
No evidence for religion?
Eric MacDonald also asserts in his review that “there isn’t a shred of evidence to show that religion is true.” I have to say he’s flat wrong on this point. If MacDonald wants evidence, I can show him some: the evidence from miracles.
The philosophical arguments against the possibility and/or credibility of miracles, have been dealt with by Dr. Timothy McGrew in his article, Miracles in The Stanford Encyclopedia of Philosophy, so I won’t waste time on them here.
Eric MacDonald will want to see good evidence of miracles, so I’ll confine myself to one case: the 17th century Italian saint, Joseph of Cupertino, who was seen levitating well above the ground and even flying for some distance through the air, on literally thousands of occasions, by believers and skeptics alike. The saint was the phenomenon of the 17th century. Those who are curious might like to have a look at his biography by D. Bernini (Vita Del Giuseppe da Copertino, 1752, Roma: Ludovico Tinassi and Girolamo Mainardi). The philosopher David Hume, who was notoriously skeptical of miracle claims, never even mentions St. Joseph of Cupertino in his writings. Funny, that.
The evidence for St. Joseph’s flights is handily summarized in an article, The flying saint (The Messenger of Saint Anthony, January 2003), by Renzo Allegri.
The earthly existence of Friar Joseph of Cupertino was rich in charismatic gifts. However, the phenomenon which attracted the most attention occurred during his disconcerting ecstasies. Chronicles recount, as we have already said, that he need only hear the name of Jesus, of the Virgin Mary, or of a saint before going into an ecstasy. He used to let out a wail and float in the air, remaining suspended between heaven and earth for hours. An inadmissible phenomenon for our modern mentality.
‘To doubt is understandable,’ Fr. Giulio Berettoni, rector of the Shrine of St. Joseph of Cupertino in Osimo tells me ‘but it isn’t justifiable. If we take a serious look at the saint’s life from a historical point of view, then we see that we cannot question his ecstasies. There are numerous witness accounts. They began to be documented in 1628, and this continued until Joseph’s death in 1663, i.e. for 35 years. In certain periods, the phenomenon is recorded to have taken place more than once a day. It has been calculated that Joseph’s ‘ecstatic flights’ took place at least 1,000 to 1,500 times in his lifetime, perhaps even more, and that they were witnessed by thousands of people. They were the phenomenon of the century. They were so sensational and so public that they attracted attention from curious people from all walks of life, Italians and foreigners, believers and unbelievers, simple folk, but also scholars, scientists, priests, bishops and cardinals. They continued to occur in every situation, in whatever church in which the saint prayed or celebrated Mass. It is impossible to doubt such a sensational and public phenomenon which repeated itself over time. It is also worth noting that these events occurred in the seventeenth century, the time of the Inquisition. Amazing events, miracles and healings were labelled magic and the protagonists ended up undergoing a trial by the civil and religious Inquisition. In fact, St. Joseph of Cupertino underwent this very fate because of his ecstasies. But he was subjected to various trials without ever being condemned; final proof that these are sensational events, but also real, extraordinary and concrete facts.’ (Emphases mine – VJT.)
In view of the fact that miracle claims can be found in many different religions, it would be imprudent to cite St. Joseph’s levitations and flights in support of any one particular religion. But miracles like this, which could be prompted by St. Joseph’s hearing “the name of Jesus, of the Virgin Mary, or of a saint,” certainly constitute evidence for religion. MacDonald may or may not be persuaded by such evidence, but evidence it certainly is. In the meantime, he might like to have a look at an article by Dr. Michael Grosso, entitled, Hume’s Syndrome: Irrational Resistance to the Paranormal (Journal of Scientific Exploration, Vol. 22, No. 4, pp. 549–556, 2008).
Investigating the Ridiculous
A true scientist is not constrained by metaphysical blinkers, such as “methodological naturalism” or “empiricism.” A true scientist is fearless, and prepared to investigate any claim, no matter how much it conflicts with our preconceptions. Discoveries in quantum physics in the 20th century have forced scientists to question our common-sense assumptions about reality and the nature of causation. That is all to the good, and we should welcome further investigations.
As an example of what I mean when I talk about open-mindedness, I’d like to mention the research of Dr. Daryl Bem, of Cornell University, whose article, “Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect” (Journal of Personality and Social Psychology, 2011 Mar;100(3):407-25) claimed to find evidence of retroactive causation. In a nutshell, what Bem is claiming is that events that haven’t yet happened can influence our behavior.
Here is how Bem describes his findings, in his online Response to Alcock’s “Back from the Future: Comments on Bem (a reply to a critical review, which appeared in the March/April 2011 edition of The Skeptical Inquirer):
My article reports nine experiments, involving more than 1,000 participants, that test for precognition or retroactive influence by “time-reversing” well-established psychological effects so that the individual’s responses are obtained before rather than after the stimulus events occur. Each time-reversed experiment tests the straightforward hypothesis that we should observe the same effect that we normally observe in the standard (non-psi) version of the experiment. Five different effects are tested in this way; and, to bolster confidence in the results, four of the nine experiments are actually replications of Bem – the other experiments in the article. Across all nine experiments, the combined odds against the findings being due to chance are greater than 70 billion to 1.
70 billion to 1. That’s pretty impressive, if it holds up. The field of parapsychological research is littered with claims that haven’t held up, but the methodology of this study is nothing if not scientific. As one of Bem’s reviewers, Professor Joachim Krueger, put it:
“My personal view is that this is ridiculous and can’t be true. Going after the methodology and the experimental design is the first line of attack. But frankly, I didn’t see anything. Everything seemed to be in good order.” (Quoted by Peter Aldhous, Is this evidence that we can see the future? In NewScientist 16:29, November 11, 2010.)
Krueger’s online article, Why I don’t believe in precognition is a model of scientific impartiality. He writes:
In a paper published by the Journal of Personality and Social Psychology (JPSP, 2011), Daryl Bem notes that 34% of psychologists in one studied sample believe that psi is impossible. I would be among those 34% if I were sampled. Yet, I remain intrigued by attempts to prove the existence of psi. Bem defines psi as “anomalous processes of information or energy transfer that are currently unexplained in terms of known physical or biological mechanisms.” …
I am not ready to trade my traditional view of how causal processes work themselves out through time for an “anything-goes” view. Bem knows that massive paradigm shifts require powerful evidence. The evidence he presents hardly clears that high threshold. On the plus side, there are 9 experiments conducted on different but related topics. On the minus side, the effect sizes are rather small; some of the null hypotheses are rejected only by way of the flat-footed one-tailed test, and finally, it took apparently 20 years and extensive pilot testing to put together this package of studies. Bem also knows that skeptics will clamor for independent replication studies. To his credit, he encourages such replications and provides the programs for running these studies.
[The] missing explanatory step is how we get retroactive causation at the macroscopic level from indeterminacy at the subatomic level.
Aldhous mentions in his article that one failed attamept at replication has already been posted online. In response, Bem has argued that online surveys are inconclusive, because one cannot know whether volunteers have paid sufficient attention to the task. Time will tell.
What I find impressive, however, is that the reviewers were fair-minded enough to urge the publication of an article which claimed to shatter our most deeply held beliefs about the nature of causality – and all in the name of scientific fairness. As Bem points out, “several of the reviewers expressed various degrees of skepticism about the reality of psi, while still urging the article’s acceptance.” Two reviewers, Chick Judd and Bertram Gawronski, were more sanguine: they wrote that Bem’s findings “turn out traditional understanding of causality on its head. A central assumption in lay and scientific conceptions of causality is that a cause precedes its effects, not the other way round [and] we openly admit that the reported findings conflict with our beliefs about causality and that we find them extremely puzzling.”
Fairness and fearless investigation. Those are two vital hallmarks of good science. Science withers on the vine when it is kept in check by metaphysical shibboleths and pundits who insist that something cannot be published because it is “impossible.”
Would it be too much to ask for the same spirit of openness in the field of biology? How long will it be, I wonder, until Intelligent Design researchers can freely publish their findings in journals like Nature?
A final quote for Eric MacDonald. “There are more things in heaven and earth, Horatio,
Than are dreamt of in your philosophy.” (Hamlet Act 1, scene 5, 166–167.)
“How long will it be, I wonder, until Intelligent Design researchers can freely publish their findings in journals like Nature?”
Precisely as long as it takes them to come up with some actual evidence.
I have just been in Shadowlands – the play about CS Lewis romance and marriage with Joy Davidson who then died of cancer. It taught me a lot about him. One was that he didn’t see prayer as asking God to intervene. That would happen whether he asked or not. It was therapeutic for him.
“I pray because I can’t help myself. I pray because I’m helpless. I pray because the need flows out of me all the time- waking and sleeping. It doesn’t change God- it changes me.”
i.e. prayer does not change what will happen – it changes the person who is praying.
Bravo VJ, great article, especially the mention of Saint Joseph of Cupertino.funny how he was also born in a manger to very poor parents. Nicknamed the flying Saint, his miracles of flight and levitation were so well known that it was considered a miracle only when he didn’t perform a miracle.
He was was always made fun of when he was younger because he was so clumsy and slow mentally, but he had such a love for the lord that he would be in complete exctacy just when thinking of him or hearing his name mentioned out loud.
He also had this incredible mastery over animals, that he commanded a pack of wild dogs to stop from attacking him.
When he first entered the monastery he was thrown out by the other monks because he kept breaking everything there. He was the least important men to the world at that time, the least significant and yet the lord worked amazing miracles.
I get so excited just thinking about his miracles. I will post some good links about him when I get to my pc tomorrow as I’m on my iPod now.
Funny how David humes never checked him out, but I guess that seeing the flying saint would have given his worldview a nightmare of a problem since humes didn’t believe in miracles.
This reminds me of the verse in the bible which talks about how God will make the smart men of this world look foolish
Of course the findings aren’t “due to chance”. Chance doesn’t cause anything. What such statements usually mean is that, assuming a random sample, the results are unlikely to be observed under the null hypothesis.
But that doesn’t mean they aren’t likely to be observed under some other alternative hypothesis than the study hypothesis.
So the issue here is not the very low “p value” but whether there is an alternative explanation for the observed results.
You say:
The methodology is certainly “scientific” but whether it is sound, methodologically or statistically, is another question.
The fact that the author has mistaken his p value for the probability of his hypothesis given his data, for the probability of his data given his null hypothesis is a red flag.
Oops, last para should read:
“The fact that the author has mistaken his p value for the probability of his hypothesis given his data, rather than for the probability of his data given his null hypothesis is a red flag.”
Ah, looks like someone has spent considerable effort on this:
http://dl.dropbox.com/u/1018886/Bem6.pdf
and essentially makes the same point as I have just done 🙂
And many other excellent points!
I’ve seen a few ‘small’ answered prayers, if there is truly anything as a ‘small’ answered prayer. One answered prayer was this:
This is one of my favorite miracles that I’ve seen so far in this life.
Elizabeth Liddle:
I (and likely others here) am genuinely interested in understanding better the basis for your ‘red flag’. I’m not arguing or criticising, but given past disagreements arising from semantics, I’m asking:
– how did Bem compute p for probability of his hypothesis given his data?
– how should Bem have computed p for probability of his data given his null hypothesis?
– how do or should the computations of p differ for:
a) probability of Bem’s data, vs.
b) probability of Bem’s hypothesis, vs.
c) probability of Bem’s null hypothesis
Well, one small problem is that Bem doesn’t actually give the details of how he computes the p value, but that’s not really my issue. What he does is misinterpret what that p value is a probability of
The p value you get from a t test, rather counter-intuitively, is not the probability that your study hypothesis is false, but the probability that you would have observed the data you did observe had your null hypothesis been true, which is not the same thing.
But (and I’ve now looked further at his paper)he makes a number of further errors, also pointed out by Wagenmakers et al.
The most egregious (and I remember noting this when the paper first came out, and made a bit of a stir) is that he doesn’t give good a priori justifications for his hypotheses, and he should make a p value correction for multiple possible hypotheses. One is the “two-tailed” test, which is absolutely standard, yet Bem says: “Unless otherwise indicated, all significance levels reported in this article are based on one-tailed tests.” But even a two-tailed test would be inadequate here, because for many of his experiments, several hypotheses could be, and were, tested (e.g. effects of gender; effects of stimulus type; effects of extroversion), and no correctionto the p value is made for these multiple hypotheses. The point being that the probability of finding some effect is much higher than the probability of finding one effect.
In other words these are exploratory experiments, not confirmatory experiments. And this raises yet another problem: you can’t do a meta-analysis on a series of exploratory studies, all of which report significant but different effects (although all supposed to support psi effects) and then claim that you have a hugely significant support for psi. You could only do that if you had a series of confirmatory experiments, all with clearly stated a priori hypotheses regarding the direction of the effect that was supposed to indicate psi.
There’s a classic article by Cohen here that you might be interested in.
So, to address your specific questions:
He didn’t. He computed p for the probability of his data given his null hypothesis, for which he used a standard t test (although he does not always make his method clear: he does not, for example, give the variance of hit-rates among his participants in experiment 1, only the mean hit-rate).
That’s what he did calculate. I still haven’t quite figured out how he did it, as he doesn’t give (as I say above) the between-subject variance in hit-rates.
Sorry I mangled the grammar there!
Null hypothesis testing tells you the probability of your data given the null. Bayesian tests can tell you the probability of your hypothesis versus some other hypothesis, but the results will depend on your priors (see the Wagenmaker et al paper). This leads some people to question the validity of Bayesian statistics, but in my view it merely makes explicit assumptions that are implicit, but not adequately considered, in classical hypothesis testing.
Classical hypothesis testing depends crucially for its validity on the quality of your null model. If you get that wrong, your results are essentially meaningless, because all you will have done is to reject something that was never likely to be true in the first place. It certainly doesn’t tell you that your hypothesis is true.
I like how one wag reported the results of the Benson study:
SCIENTISTS FAIL TO MANIPULATE GOD
Anyone conducting a clinical study on the efficacy of prayer has failed to grasp the concept of God.
No, you have it backwards. Anyone claiming that that there is clinical evidence for the efficacy of prayer has failed to grasp the nature of clinical evidence.
God may well work in ways that are invisible to scientific data analysis.
But in that case you cannot claim that there is scientific evidence for His work. It can only ever be anecdotal.
Thank you. My understanding of statistical techniques is shallow. It will take me days to crawl thru this, but thanks for the pointer.
In the same vein anyone claiming that there is clinical evidence for the efficacy of accumulations of random mutations has failed to grasp the nature of clinical evidence.
Random Mutations may well work in ways that are invisible to scientific data analysis.
But in that case you cannot claim that there is scientific evidence for “their” work. It can only ever be anecdotal.
One interesting thing I’ve just thought of: Bem proposes that there will be inter-subject differences in psi. This means that if his hypothesis is true, low psi subjects will score around 50% for a 50% probability correct task, whereas high psi subjects will score more.
So one test of his hypothesis would be to see whether a) variance in scores is greater than would be expected under the null (no psi in anyone0 and b) whether the scores skew positive (a few high scores).
tbh even if psi was a very rare ability, one person who consistently scored a lot better than chance would be more convincing than 100 people with a mean slightly higher than chance score.
So much depends on your null!
Elizabeth Liddle
And this raises yet another problem: you can’t do a meta-analysis on a series of exploratory studies, all of which report significant but different effects (although all supposed to support psi effects) and then claim that you have a hugely significant support for psi. You could only do that if you had a series of confirmatory experiments, all with clearly stated a priori hypotheses regarding the direction of the effect that was supposed to indicate psi.
A follow-up question on what is deemed accepted methodology?
Bem reported on multiple tests, ostensibly to preempt understandable criticism that a single test and result might be biased, whereas multiple tests and results show a convergence.
Yet you are flagging that same effort to use multiple, different tests as statistically incorrect.
So, what would be the statistically valid methodology to show convergence of multiple tests (assuming the underlying reality is convergent)?
[heh, just noticed another error: “The difference between erotic and nonerotic trials was itself significant, tdiff(99) = 1.85, p = .031, d = 0.19”.
This isn’t valid because there are more nonerotic trials overall, so, under the null, you’d expect the mean hit rate to be closer to .5.]
Well, meta-analysis can be perfectly valid, but one big problem is that, like all statistical analysis, it assumes a random sample from some larger (in some cases theoretical) population. So if the data going into the meta-analysis are exploratory, rather than confirmatory, data, you start off with a biased sample.
What would be valid would be to take the results of a series of confirmatory studies, each perhaps of small effect, and determine the chance of all those studies producing a net result in favour of psi.
Alternatively, one well-conducted confirmatory study showing a statistically significant effect.
Meta-analyses often weight the studies by the quality of the study. Perhaps counter-intuitively, smaller, less statistically powerful, studies tend to yield larger effects.
So while meta-analyses can be useful for establishing real, but small, effects, not all meta-analyses are created equal. What is needed are good, truly confirmatory, studies.
You forgot selection. Again.
Hi markf,
Thank you for your post, and Happy New Year, by the way. It must have been quite exciting taking part in a play on C. S. Lewis. (How exactly were you involved, if you don’t mind my asking?)
I have to respectfully disagree with your interpretation of C. S. Lewis’ thoughts on prayer. You quote him as saying:
from which you proceed to draw the inference:
But that’s not what Lewis said. What he said was that prayer does not change God. He did not say that prayer does not change outcomes. Actually “change” is not quite the right word, as God’s supernatural causation (in response to our prayers) is timeless. It would be better to say that our prayers affect outcomes, and that this happens because if we pray, God will (timelessly) decide to do something (e.g. help someone in a special way) which He would not have done if we had not prayed.
I can produce some evidence from C. S. Lewis’ own writings that shows that he really did believe that prayer changes outcomes, although of course, it cannot change God, because He is timeless. As for God’s “intervening”: this is a loaded term, because it properly applies to a time-bound being, which God is not. We can however, meaningfully speak of God as doing something because an individual prayed, which He would not otherwise have done. Or as Lewis puts it in his essay, “The Efficacy of Prayer” :
That’s an interesting way of looking at it, don’t you think?
Here is another ‘small miracle’:
further notes:
Here is one:
Actually, he does say how he computed that p value: a one-sample t test (he tells us during Experiment 2).
The binomial test (which he also uses) would be better though.
And we really need to know those distributions.
corrected link:
Then again we can attribute everything in the universe, even the universe itself, to the final cause of a ‘random miracle’ as atheists insist we do:
Further notes:
You forgot that “selection” isn’t selection but is just a result of three processes-> it is differential reproduction due to heritable random variation.
You people think that natural selection does something and everyone else finds that hilarious.
Mutations accumulate for a number of reasons and it appears the neutral theory explains more genetic diversity than NS ever can.
Perhaps you should take a refresher course in biology.
Response from Bem to Wagenmakers et al here:
http://dbem.ws/ResponsetoWagenmakers.pdf
He clears up at least some concerns.
NS doesn’t explain genetic diversity, it explains adaptation
It reducesgenetic diversity.
Perhaps you should take a refresher course in biology.
Also, there is nothing in our foundational understanding of reality that prevents miracles from happening, and in fact, there is much that argues forcefully for miracles to be expected:
Leading quantum physicist Anton Zeilinger has followed in John Archibald Wheeler’s footsteps (1911-2008) by insisting reality, at its most foundational level, is ‘information’, which corresponds to what was ‘predicted’ from John 1:1
Moreover consciousness is found to be more foundational to reality than information is:
The argument for God from consciousness can be framed like this:
references:
Here is the key experiment that led Wigner to his Nobel Prize winning work on quantum symmetries:
i.e. In the experiment the ‘world’ (i.e. the universe) does not have a ‘privileged center’. Yet strangely, the conscious observer does exhibit a ‘privileged center’. This is since the ‘matrix’, which determines which vector will be used to describe the particle in the experiment, is ‘observer-centric’ in its origination! Thus explaining Wigner’s dramatic statement, “It was not possible to formulate the laws (of quantum theory) in a fully consistent way without reference to consciousness.”
James 4: 2b You do not have, because you do not ask God. 3 When you ask, you do not receive, because you ask with wrong motives, that you may spend what you get on your pleasures. [NIV]
(SGD: Someone who, absent some pretty powerful answers to prayer over decades, would not now be here.)
NS explains the genetic diversity between diverged lineages- as in why there are different species.
Dawkins calls it cumulative selction.
What is your degree in again?
No. That was the mistake Denton made.
NS explains adaptation. Not genetic diversity. Genetic diversity is required for adaptation, not the other way round.
1- What mistake did Dention allegedly make?
2- Natural selection allegedly explains the earth’s/ ecosystem’s biodiversity. And guess what? It is linked to the genetic diversity.
3- Adaptation = whatever works “good enough”
Now, back to what I said and your obvious misunderstanding:
NOTE: accumulations of random mutations– Mutations accumulate in many ways and selection is just one of them meaning what I said INCLUDES selection and therefor, contra Elizabeth, I did not forget it as she claimed.
OK Elizabeth- did you get that, too?
Of course, if God answers prayers, there is the burning question of what he’s got against amputees.
vj
I played the part of an atheist sceptical friend and colleague of Lewis – a professor Riley – rather appropriate I think 🙂 I cannot find any reference to a real person of this description.
You are right. Obviously Lewis also thought prayer could change things. I was struck while doing the play that the emphasis was on prayer as a sort of therapy – but the article shows that he really thought it might change things – although not the cancer which killed Joy Davidman.
Hi Elizabeth,
Thank you for your posts. Just a few quick comments:
(1) Bem’s paper was carefully reviewed before it was published. As one reviewer, Professor Joachim Krueger, put it:
(2) It seems to me that the methodological and statistical criticisms you raised in your comments were adequately answered by Bem in his response to Wagenmakers et al. at http://dbem.ws/ResponsetoWagenmakers.pdf
(3) I’ll just quote a few comments from Bem’s response:
In other words, as long as you’re prepared to believe that the prior probability of psi existing is 0.00000001, then you’ll be convinced by Wagenmaker’s analysis, but if you’re disposed to believe that the probability of psi existing is only 0.0000000000000000001, then of course, Wagenmaker’s findings won’t convince you.
(4) This brings me to my central concern with Bayesian statistical testing: by setting your prior probability low enough, you can guarantee that nothing will convince you. For instance, the total number of events in the entire history of the observable universe has been estimated by Seth Lloyd at around 10^120. If you set your prior probabilities low enough, you can ensure that the total number of tests that would have to be performed in order to change your mind would be greater than the number of events in the history of the universe – and hence guarantee that nothing will.
(5) Wagenmaker et al.’s arguments against psi at http://dl.dropbox.com/u/1018886/Bem6.pdf are fairly weak anyway: Darwinism (yes, but even if you’re a theistic evolutionist like Professor Ken Miller you could still find room for a soul) and lack of success at casinos (which only proves that if we have psychic powers, they don’t pick up anything and everything. Most people are pretty bad with numbers anyway, so I’m not surprised that they’re bad at picking up numeric information.
(5) Heck, I wouldn’t even put the probability of my own non-existence at 10^-20, let alone the non-existence of psi 🙂
Cheers.
Hi bornagain77,
Thanks for sharing your very moving story.
Being a bit lazy here – but did Bem just multiply the p values together to get a total p value?
He used Stouffer’s Z.
1. Yes, Bem’s paper was carefully reviewed, and published in a good journal. That doesn’t mean it’s immune to criticism.
2 &3. Wagenmaker’s Bayesian Factor estimates do not depend on his priors about psi, but on a default distribution for Bem’s effect sizes. Bem disputes that default, and he may be right. But he also may be wrong, and that is the point. His p value doesn’t tell you that he is right, merely that his data are unlikely to be observed under his null. It’s his null that is at issue.
4. Your concern is reasonable, but that’s not a problem specific to Bayesian methods. Bayesian methods merely make the issue explicit. Classical statistics don’t obviate the problem, it merely makes its conclusions unreliable. In both cases, the issue concerns the appropriate null. In Bayesian methods you cannot proceed until you have explicitly formulated it. In classical statistics it’s all too easy to assume you have an appropriate null when you haven’t.
5. Those are not Wagenmaker’s only arguments; and they have force. Without a theoretical basis for a hypothesis it is very difficult to construct an appropriate null, and it still leaves open the issue that if a phenomenon is highly unlikely, even an unlikely result is more likely to be the result of experimental or analytical error than the phenomenon you have proposed. Which is exactly why Bayesian methods are so important in medicine – why, counter-intuitively, a rare positive test on a screen for an even rarer disease still doesn’t mean you are likely to have that disease.
5 again :D) Too much statistical nerdery rots the brain 🙂
vjtorley, I think Bem’s paper is intriguing. But, as Wagenmakers et al, and indeed one of his reviewers, say it is completely atheoretical, and so the first question has to be: what more mundane explanation could there be for the data? And the writing of the paper does not fill me with confidence. His use of t tests, for a start. While he also backs these up with binomial tests, he completely fails to note the implications of his own findings of absent apparent psi effects in some individuals. That has direct implications for the distribution of hit rates in his sample (should make them strongly positively skewed). But he doesn’t even mention the variance in hit rates and appears to show no interest in that distribution, except to say that the normality assumptions of the test might be violated. And yet he’s quite happy to use a one-tailed test on the principle that a psi effect can only work positively. Why? In the absence of a theoretical basis you have to use a two-tailed test. The only justification for a one-tailed test is when you know that your negative tail must be noise. And he makes othe errors two (like that test of the difference in success-rate between erotic and non-erotic stimuli, ignoring the fact that there were, overall, more non-erotic trials, and therefore less chance, under the null, that the hit-rate would be much higher than 50%. Incidentally, for the heck of it, I ran a quick simulation, under the null hypothesis, and of course found that consistently, the “erotic” stimuli had more extreme mean scores than the “non-erotic”.
That error really shouldn’t have got past the reviewers.
I haven’t yet been through the whole paper with a fine-toothed comb, but that’s my haul so far 🙂
But if he’s right, all that is required is a replication, by a different team, with the same protocols, and appropriate increased sample sizes. Even a replication of Experiment 1, possibly with all erotic stimuli, would be very impressive.
erratum: “other errors too”.
Errors about errors!
Nothing. There are several incorrect assumptions you are making here. First, you assume that Christians can dictate what God should do. We serve him, not vice versa. Two, you assume that God simply answers all prayers. He does not. He pick and chooses what to allow and not allow based off all that infinite knowledge he has. Three, you assume that there is a person who “deserves” it. God is clear, there are no people who are inherently worthy. All have sinned. Four, I am going to go out on a limb and say you are atheist? Well, from your perspective this life is the one that matters most. To a Christian however after this life is a paradise where all wounds are healed. What is 60 or 70 years of being limbless compared to eternity?
God has nothing against amputees. He is just the sovereign and may have a better idea of how to run things than you.
As to the question, Why doesn’t God heal amputees? ,,, If you find a answer for that question, I would like for you to also ask God, since only God knows his entire purposes for when He allows divine miracles, why is the death rate still 100% for humans since He, as Jesus incarnate, defeated death on the cross?,,, But as to your primary premise in your argument that God NEVER has healed a amputee, we find:
Well, there was the guy whose ear one of Jesus’ followers cut off, and Jesus touched it and it was healed.
(Luke 22:51)
But Jesus answered, “No more of this!” And he touched the man’s ear and healed him.
I don’t know if you call that an amputee, though I sure would!
My main point with the ‘death question’ though is so what if there were documented cases of amputees being miraculously cured. Then you would say there were no cases of multi-drug-resistant TB being cured miraculously. Or no cases of autism cured miraculously. Or no cases of bird flu cured miraculously. Or no cases of mad cow disease cured miraculously. Seems like a weak argument to me. Indeed seems like a pathetic ‘excuse’ for not believing in God rather than acknowledging the overwhelming evidence for God’s existence that modern science has revealed. The true reason you don’t believe in God is because you don’t want to believe in God period!!! i.e. The evidence for God’s reality, from science itself, is overwhelming, whereas the ‘excuses’ for not believing are contrived rationalizations!!!
Timbo don’t be bimbo.
Bornagain, thank you very much for touching stories.
God wants his miracles to be indistinguishable from natural events, just as the Intelligent Designer (could they be one and the same?) wants his designs to appear as if they evolved.
That’s a common thought, but it means that science can proceed on its present course without worrying about interventions. It means the most important assumptions of the scientific enterprise — continuity and regularity — are fully justified.
Indeed, he’s so careful about covering his tracks that he might as well not exist — and probably doesn’t. But if he does exist, he must be annoyed at the pesky ID proponents who keep trying to flush him out of hiding when he would obviously prefer to remain incognito.
,,,Only in Theism are they, continuity and regularity, fully justified! Theism alone presupposes, in its foundational beliefs, transcendent universe laws that are unchanging. Transcendent laws that were put in place by the rational mind of God that can be discovered by the rational human mind of man who was created in His image. In fact Atheism demands ‘randomness’ as a cornerstone premise in its foundational view of reality and thus atheism has no right to presuppose that the transcendent universal laws, that have been discovered, should be unchanging, much less did the atheistic worldview presuppose their existence in the first place!:
Moreover the atheistic worldview cannot even justify ‘doing’ science in the first place:
This following site is a easy to use, and understand, interactive website that takes the user through what is termed ‘Presuppositional apologetics’. The website clearly shows that our use of the laws of logic, mathematics, science and morality cannot be accounted for unless we believe in a God who guarantees our perceptions and reasoning are trustworthy in the first place.
This ‘lack of a guarantee’, for trusting our perceptions and reasoning in science to be trustworthy in the first place, even extends into evolutionary naturalism itself;
The following interview is sadly comical as a evolutionary psychologist realizes that neo-Darwinism can offer no guarantee that our faculties of reasoning will correspond to the truth, not even for the truth that he is purporting to give in the interview, (which begs the question of how was he able to come to that particular truthful realization, in the first place, if neo-Darwinian evolution were actually true?);
Atheism simply is bankrupt of any substantiating evidence from science itself, nor for ‘why’ we should even be able to do science in the first place. Moreover it is uniquely the Christian worldview, and none other, which brought science to maturity in the first place:
Well He certainly did not ‘hide’ from these guys:
You know what I’ve been noticing lately? I never need to look at the author of a vjtorley post to know it’s vjtorley.
Thanks for having such provocative and substantive contributions Dr. Torley. 😀
Hi Elizabeth,
I appreciate your concern for sound statistical methodology. I won’t argue with you on whether a one- or a two-sided t-test would have been more appropriate here, except to note that Bem claims in his response to Wagenmakers et al. that even if a two-sided test were employed, the results still constitute evidence in favor of the psi alternative, with a posterior probability on the composite H0 of 7.3 × 10^-5.
Incidentally, I came across an article by Dr. Joachim Krueger in Psychology Today (January 19, 2011) in response to the critiques of Bem’s paper by Wagenmakers et al. and also by Rouder and Morey . Krueger thought that these authors were misusing Bayesian methodology, to mask their deep-seated prejudices. Commenting on one passage by Rouder and Morey, Krueger wrote:
I also notice that Rouder and Morey , unlike Wagenmakers et al., used a one-sided prior weights (performance must be at or above chance), as they considered this to be consistent with Bem’s analysis.
I will agree with you about one thing, Elizabeth: future testing definitely needs to incorporate increased sample sizes. Why not test thousands of students at once? It could be done. Bem’s study, while intriguing, is hardly conclusive. We need follow-up studies.
Frankly, I’m doubtful about Bem’s claimed effect, and if I were a betting man, I’d bet against it. Still, I might be wrong, and I’m prepared to consider the possibility of retroactive causation. It is a hypothesis which needs to be examined, and not ruled out of court.
(By the way, the late philosopher Michael Dummett was a great defender of retrospective prayer, as you’re probably aware. Recently, Kevin Timpe wrote a very interesting and highly readable paper on the subject, entitled, Prayers for the past, in which he considers Dummett’s view.)
Yes, I’d bet against it too, but I certainly wouldn’t rule it out.
However, the fact that Bem made some actual statistical errors leads me to think it more likely (loads my priors) that the effect is due to a methodological glitch than psi. And Wagenmaker’s casino point is a good one (as, actually is his Darwinian point): if psi were common enough that you can find a significant effect in only 100 volunteers, then why aren’t roulette tables losing hand over fist? The task was, in effect, a roulette task, and the house effect on a roulette table is smaller than the psi effect in the study.
To do psi research i.e. research in a field where there is no theoretical mechanism at al and where the field is rife with fakes and wishful thinking, your statistical techniques really need to be whiter than the driven snow, and Bem’s weren’t.
I’d take issue with the allegation that Bayesian stats can be used to “mask deep-seated prejudices”. On the contrary, they require you not only to unmask your deep-seated prejudices but to put a number on them (as Krueger agrees). Wagenmakers et al’s BFs don’t mean that psi is false. They do mean that Bem’s p values need to be taken with a strong pinch of salt. A classical p value simply doesn’t mean what it is tempting to think it means.
Interestingly, I had bitter experience (well interesting, but bitter too!) of this in 2004, when Bush beat Kerry, and there was a lot of internet chatter about the exit polls which had shown Kerry beating Bush, polls with a p value of some infinitessimal probability of being “due to chance”. Ergo, Bush stole the election.
Of course the exit polls weren’t “due to chance”. But “Rove stole it for Bush with electronic voting machines” was not the only alternative hypothesis.
In this case, it’s not clear what the alternative hypothesis is. But the solution is easy enough, as I said, and you agree – independent replication, announced in advance, and powered to find a smaller effect with high probability.
BTW, for what it’s worth, I ran a Monte Carlo simulation of Bem’s Experiment 1 and I agree that his effect is significant for “erotic” stimuli, even if we correct for two hypotheses. However, that doesn’t mean that psi is real 🙂
I am wondering what to conclude if the correlation does prove to be robust. Correlation is not causation – particularly backwards causation! Something odd is going on – but an unknown causal relationship going the conventional way seems more plausible than an unknown causal relationship going backwards.
Surely that is true of most posters?
For example:
vj: Small book, well written and documented, but a bit weird
news/denyse: gossip column
gil: generic pronouncement on the irrationality of atheist/materialists – generally linked to his own experience
cornelius: religious nature of science
Elizabeth Liddle:
And Wagenmaker’s casino point is a good one (as, actually is his Darwinian point): if psi were common enough that you can find a significant effect in only 100 volunteers, then why aren’t roulette tables losing hand over fist?
Because gaming houses (e.g. Vegas casinos) are in business to keep their winings and their odds. They routinely as a matter of course identify and physically remove people who win more then the house figures they should have. If psi is in fact a genuine phenomena, it will not be tolerated by the gaming industry (and they won’t be publicising how their odds were beaten either), just like they don’t tolerate any but the occasional big winner (because that’s good for business in the long run).
Their business model is geared toward losers, excluding consistent winners, and they’ll keep it that way.
This kind of testing went on at Duke University for decades, and the only result was that the effect diminished as controls became more rigorous.
Fair point, Charles 🙂
I’m not going to concede it’s a fair point. Casinos exclude people who count cards in games where skill at knowing what cards are left in the deck makes it possible to win. I’m not aware of anyone who has ever been excluded from a game of “pure chance.”
There are, of course, people offering to teach you (for a price) how to win at games that appear to be pure chance. I have no idea whether any of them have sound strategies.
It is, of course, somewhat hypocritical to exclude players on the basis of skill.
Something that needs to be considered in any test of psychic ability is the tested person’s ability to apply non psychic skills. Call it the Monty Hall effect or Monty Hall paradox. It is very difficult to design a test that cannot be beaten.
It took decades at Duke University to reach this conclusion.
This is a small video part of the reluctant saint (saint joseph of cupertino).
He was seen levitating and flying by many many people in his village and even did it for people outside his village. There is enough evidence to believe in God but not enough where you are forced to . If you want to deny something badly enough you can argue down into a spiral on almost anything. Can I be 100% scientifically sure that My dad loves me? Impossible. Do I trust that my Dad loves me purely and from all his heart. Absolutely.
http://www.youtube.com/watch?v=_rQZC7pGT-4
Its also funny that atheists want 100% proof of God. IF God gave us 100% proof of his existence would we need faith?, and when the topic of the shroud of Turin keeps coming up notice how most atheists abandon their belief in Science and then turn to loony conspiracy theories that have no scientific evidence backing them up. Some will say That you cant be sure that Jesus’s resurrection caused that Image on the shroud, but I say who else could have caused it logically and reasonably as well as historically.