Medicine News Peer review

Peer review: Dodge n’ weave: How research results are jazzed up…

Spread the love

… by hyped claims that the data do not support. From Scientific American:

This kind of academic spin is common in abstracts: in a past study, she [Isabelle Bouton] and her colleagues found spin somewhere in the abstracts of 40% of trials. And it follows through into about half of press releases and subsequent media coverage.

Today she told us about a trial they did to see whether readers get caught out by the spin. They re-wrote abstracts to take out the spin and randomized 300 people to get either the spun or cleaned-up version. Not surprisingly, they found that the spin was successful in leading readers to believe the intervention was more beneficial than it was.

You’ll be glad to learn that this concerns medical science. );

29 Replies to “Peer review: Dodge n’ weave: How research results are jazzed up…

  1. 1
    Andre says:

    Link is not working for me, perhaps Nick Matzke put pressure on them to retract it?

  2. 2
  3. 3
    sigaba says:

    So now I can ignore all those Science Daily links people post, right? Those are just glosses of the abstracts, after all…

  4. 4
    News says:

    Sorry, Andre, link now fixed. One problem when stories are scheduled rather than posted live is that if links don’t work, it is difficult for the poster to know. So thanks much for telling us.

  5. 5
    News says:

    Some of us find ScienceDaily links instructive despite the hype. It’s like hearing noise and seeing smoke. Something happened, but what? One looks for patterns in the events.

  6. 6
    Andre says:

    The evo papers are the most fun, look for words

    thought to be

    presumed to be

    believed to be

    need more data

    astonished

    when you see those words let your balcony detector go off because it ain’t science….

  7. 7

    Actually, your first four I would expect to find in any scientific paper, because scientific conclusions are always provisional, and must always have caveats. This is simply good practice.

    And any good finding generates further predictions that need to be tested on new data. So we always need more data.

    I’ve never seen the last one in a scientific paper.

  8. 8
    Andre says:

    Dr Liddle, that is of course just 3 simple examples but when you do the next read on a peer review paper make a mental note on the word…. it is everywhere!

    “100 Amgen scientists were astonished to find that they were able to replicate the results of only 6 of 53 widely cited landmark cancer research papers.”

    http://www.ahrp.org/cms/content/view/839/9/

    “Stanford scientists eavesdrop on erupting volcano’s astonishing seismic sound”

    http://news.stanford.edu/news/.....71413.html

    “Peering to Edge of Time, Scientists Are Astonished”

    http://www.nytimes.com/1989/11.....038;src=pm

  9. 9
    Andre says:

    Dr Liddle

    How is anything that is thought to be, presumed to be or believed to be ever going to be fact? Science then is not based on facts? but on thoughts presumptions and beliefs?

  10. 10

    Andre: strictly speaking, Andre, there are no facts in science. What we have are data and models and even our data are models at a lower level of analysis.

    Science consists, I suggest (and I’m not alone!) of fitting models to data. Where these models fit the data extremely well, and the models are independently confirmed by consilient models of other data, we informally regard them as facts. But all “facts” in science are provisional, and come with the either explicit or implicit caveat that future models may modify them.

    That does not mean that everything is up for grabs. The extraordinary thing about science is that over the centuries our body of “facts” – models that are overwhelmingly and consiliently supported by data – have give us a mega-model of the world that is extremely powerful at explaining – predicting – all sorts of things. It is this mega-model that has enabled us to put men on the moon, robots on Mars, and re-engineer the molecules of life.

    So you can call this a “body of knowledge” if you like. But it is all still a model, and the sub-models of which it consists are still fluid and subject to continual tweaking and revision.

  11. 11

    Andre, those are news headlines, not scientific papers.

  12. 12
    Andre says:

    I’ll go through the mountain of papers I have and point the peer review ones out using the word astonishing….. is that ok with you?

  13. 13
    Andre says:

    Dr Liddle

    Lots of spin in your post #10 but really no answers…… So can science be factual or not? When is something a fact? Maybe you don’t know what fact really means?

    Fact: http://en.wikipedia.org/wiki/Fact

    Lastly Dr Liddle, science is about what causes what…. its not complicated its not mysterious and if we don’t know we don’t know. But there is much that we do know!

  14. 14

    Andre:

    I’ll go through the mountain of papers I have and point the peer review ones out using the word astonishing….. is that ok with you?

    Yes, sure, Andre. I’d be interested.

    @ 13 No spin there at all Andre. Whether science is “factual” or not depends on how you define a fact. It’s a word I avoid because I don’t think it belongs in science. But informally, an extremely well supported model can be regarded as a fact. For example, we regard it as a “fact” that the earth is spherical, not flat, even though it is not precisely spherical. Spherical is a pretty good model for most purposes, as the fit of model to data is extremly tight.

    Your wiki link endorses this:

    “Scientific facts are verified by repeatable experiments.”

    Precisely. If a model is verified by repeatable experiments, we regard it as a fact. However, scientifically, no fact is inviolable, which is why I avoid the word. By that wiki definition, a fact is an extremely well-supported conclusion from many experiments. But all conclusions in science are provisional, and must be.

    Lastly: I disagree: science is about what predicts what, but it is not necessarily about causality. We can often infer causality, but not always. The underlying reality may simply be a set of commutative equations, where we cannot tell what the first mover is, and indeed may conclude that such a question is meaningless – or indicates a deity. But such conclusions would be metaphysical, not scientific.

  15. 15
    Andre says:

    Dr Liddle

    Who is even speaking about the first mover? You are and it’s because the idea of a first mover obviously frightens you! In this universe we are after causes… so science is WHAT causes WHAT? unless it is repeatedly demonstrated what the cause is it might be beliefs, and assumptions but it is never a fact!

    This brings me to the point of your treasured subject evolution. Dr Larry Moran gives the best explanation for evolution and this is about as close to fact as you’ll get

    “Evolution is a process that results in heritable changes in a population spread over many generations.”

    Everything else bolted onto that is not a fact and is in fact not even science.

  16. 16

    The idea of a first mover doesn’t frighten me at all, Andre.

    All I’m saying is that science is not exclusively about
    causality. It’s about covariance. Sometimes we can infer causality from covariance, sometimes we can’t. But not knowing which lever you press to make the other stuff move doesn’t alter the fit of the model.

    Your quote from Larry Moran is not a “fact” but a definition.

    Definitions are important so that we know what we are talking about, but they are not scientific “facts”.

  17. 17
    Andre says:

    Dr Liddle

    I know its a definition, I did not say the definition is a scientific fact, what we observe from the definition is as closer to a fact as you can get.

  18. 18
    Barb says:

    Dr. Liddle @ 14: “Whether science is “factual” or not depends on how you define a fact. It’s a word I avoid because I don’t think it belongs in science.”

    “Today, the theory of evolution is an accepted fact for everyone but a fundamentalist minority, whose objections are based not on reasoning but on doctrinaire adherence to religious principles.” – James D. Watson

    Scientific understanding requires both facts and theories that can explain those facts in a coherent manner. Evolution, in this context, is both a fact and a theory. It is an incontrovertible fact that organisms have changed, or evolved, during the history of life on Earth. And biologists have identified and investigated mechanisms that can explain the major patterns of change.” – Richard E. Lenski (http://www.actionbioscience.or.....enski.html)

    A couple of your fellow scientists diagree with you. And this illustrates why I believe that science does not hold ultimate authority, at least not in the way scientists like Steven Pinker want it to. It’s because scientists can’t agree amongst themselves what the truth is; one scientist says facts don’t belong in science, two others disagree.

  19. 19
    Andre says:

    Dr Liddle

    Please tell me what the heck does the measure of two random variables have to do with scientific facts? We’re not talking about statistics and probability theory.

    There are no probabilities, or statistic when I drop sodium in water……

  20. 20

    Well, Andre, as I said, a “scientific fact” is a model so well supported by evidence that we can regard it is a fact.

    It is still a model.

    All science boils down to probabilities, and that includes what happens when you drop sodium in water. However, we can predict, with an astronomically high degree of confidence, what will happen when you do, although that prediction will have to include a fair number of factors such as the temperature of both the sodium and the water.

  21. 21

    They don’t really disagree, Barb. As I said, extremely well supported models can be regarded as facts. In fact, if you define a “fact” as in wiki, then that is the definition a fact.

    The important thing to remember, if that is how you are using the term, is that all facts in science are still models

    Science is not a process of uncovering facts and then stacking them up, never to be questioned again. All facts are still models, it’s just that once a model is supported by extremely good evidence, you don’t have to go re-supporting it all the time. You can regard it as a fact and move on. Just remember that your subsequent conclusions are only as good as those facts, and they are still model.

  22. 22
    kairosfocus says:

    EL: In 10 above you slid down a slippery slope from factual data being “facts” to explanatory models being deemed facts — essentially based on consensus. This is a gross error. Empirical facts test theories, and in a great many cases, facts are essentially certain, or are effectively certain within a set of reasonable measurement error bars, e.g. g = 9.8 N/kg or c = 2.99+ *10^8 m/s or e = 1.602 * 1-^-19 C, etc etc, just consult the CRC rubber bible for a massive collection. Similarly, no reasonable person will dispute the factual status of the existence of Teddy Roosevelt, or Geo Washington, or Napoleon, or John Knox, or or Genghis Khan or Augustus Caesar or Jesus of Nazareth [cf. here on FYI Mr Dawkins, et al], or Confucius, etc etc etc. of course, warrant for facts is not to be confused with absolute deductive proofs, but moral certainty on adequate experience or observation can amount to essentially absolute status in a great many cases. And of course there are self evident and undeniable truths, on pain of absurdity. Explanations account for facts, they are not equivalent to them and should not be confused or conflated with them. Which, I know is a particular problem in evo mat origins science and science edu. KF

  23. 23

    There isn’t a “slippery slope”, KF, because it is precisely calibrated. “Facts” are simply explanatory models with an extremely good fit to data. It isn’t “based on consensus”, it’s based on math. We can quantify how well a model fits the data. We can regard the great equations of physics as facts, because the amount of gap between their prediction and our observations is vanishingly tiny.

    So I am actually agreeing with you (or you with me). Facts in science are not determined by “consensus” but by model fit.

    Your other facts, for example the existence of a US president called George Washington, are not scientific facts in that quantitative sense, but the same principle applies. That Washington was president is supported by a vast amount of consilient data. That he chopped down a cherry tree is not.

    And, as you probably know, frequently, previously assumed “historical facts” often cease to be regarded as “facts” when new evidence comes to light. Currently in the UK there is lively renewed debate about whether Richard III really had the Princes in the Tower killed or not, something I was once taught as a historical “fact”. It may well be true, but it no longer has the status of a historical “fact”. Even in the humanites, established “facts” are open to revision in the light of infirming data.

  24. 24
    kairosfocus says:

    EL, pardon but you are in manifest error. And that there may be errors of history or issues and doubtful claims does not detract from the manifest point that while claims are open to correction in principle, there are morally certain facts that in many key cases are essentially absolutely certain. Do you care to deny that there was a Second World War, or a Shoah sadly connected thereto, or a Roman Empire, etc? It is looking a lot like you want to open the door to dismissive selective hyperskepticism against what you do not wish to accept, conjoined to confirmation bias on those things that you wish are so per a priori ideological frame. KF

  25. 25

    KF, you are misreading me. I’m saying nothing different from what you are saying.

    If something is so probable that we can be “morally certain” of it (whatever that means) then we don’t need to worry about whether it isn’t. If it’s a little less probable than that, then we need to be a little more cautious.

    Of course I’m not denying that there was a Second World War. I’m not even asking that we be skeptical about extremely well-fitting models. I’m just pointing out that scientific knowledge does, nonetheless, consist of models.

    This is really quite uncontentious.

    Of course I don’t “want to open the door to dismissive selective hyperskepticism against what you do not wish to accept”. It’s precisely that kind of selective hyperskepticism that I accuse IDers of!

    We have a perfectly good metric for estimating how much confidence we can put in a scientific model, and in its various submodels. That’s why, as you point out, we have error bars. When those error bars get so tiny we can’t see them, then we have no cause for “hyperskepticism”.

    It’s just a waste of time. That’s why there is no serious debate about the age of the earth. There are error bars on the estimate, for sure, but they get nowhere near to embracing 10,000 years.

  26. 26
    Mung says:

    Elizabeth Liddle:

    strictly speaking, Andre, there are no facts in science. What we have are data and models and even our data are models at a lower level of analysis.

    The Fact of Evolution

    Evolution as fact and theory

    Is Evolution a Theory or a Fact?

    Theory and the Fact of Evolution

    Wake up, Lizzie.

  27. 27

    Why don’t you read my whole post, Mung?

    This perennial quote-mining is getting tiresome.

  28. 28
    Mung says:

    Elizabeth:

    Why don’t you read my whole post, Mung? This perennial quote-mining is getting tiresome.

    I’d love to quote you contradicting yourself. Is that what you want?

    Elizabeth Liddle:

    Where these models fit the data extremely well, and the models are independently confirmed by consilient models of other data, we informally regard them as facts.

    There are no facts, but there are facts, but facts are only facts “informally.” And it’s the models that are facts, even if only “informally.” Because that’s what you said. It’s nonsense, but it’s what you said. And you’re the one who insisted that I quote you fully.

    I read your whole post Elizabeth, but I chose not to expose the full nonsense. But since you insisted.

    Let’s continue, lest I be accused of quote mining:

    The extraordinary thing about science is that over the centuries our body of “facts” – models that are overwhelmingly and consiliently supported by data – have give us a mega-model of the world that is extremely powerful at explaining – predicting – all sorts of things.

    So the facts are the models. Not the data, it’s not the data which is factual, it’s the models.

    This is your stated position. Do you deny it?

    So when you say there are no facts, only models, and it’s the models which are the facts, you are both affirming and denying the existence of scientific facts.

    Your stated position is incoherent.

  29. 29

    Well, it seems pretty clear to me Mung. I’m sorry you find it incoherent.

Leave a Reply