Don’t Trust Computer Simulations And Models That Can’t Be Tested Against Reality

Following up: This excerpt is also useful food for thought, on how computer modelling has diverged from insightful mathematical modelling, with what implications:

Before cheap, large, fast computers existed, "mathematical modeling" was indistinguishable from the construction of mathematical "theories" describing particular phenomena. Calculations were commonly done by scientists who had a grounding in differential and partial differential equations—a grounding that was often based on fluid dynamics, electromagnetic theory, Schrödinger's equation and the like. Those scientists (like their counterparts today) were familiar with a wide range of approximate and asymptotic methods . . . . With modern computers, it is now possible for a graduate student or a practicing engineer to acquire a very complex computer code, hundreds of thousands of lines long, worked over by several preceding generations of scientists, with a complexity so great that no single individual actually understands either the underlying physical principles or the behavior of the computer code—or the degree to which it actually represents the phenomenon of interest. These codes are accompanied by manuals explaining how to set them up and how to run them, often with a very long list of "default" parameters. Sometimes they represent the coupling of two or more submodels, each of which appears well understood, but whose interaction can lead to completely unexpected behavior (as when a simple pendulum is hung on the end of another simple pendulum). One hundred years in the future, who will be able to reconstruct the assumptions and details of these calculations?

How many people really understand the latest climate models in depth, line by line, module by module, interaction by interaction,a nd globally? Evolutionary models? Econometric models, etc etc? How well are these calibrated against material, well warranted data sets? What track record of successful prediction obtains, and what are the environamental constraints that may invalidate such predictive power? [Known unknowns? Unknown unknowns? . . .] And more . . . Maybe we need to do some serious rethinking? "Mirror, mirror on the wall . . .?" GEM of TKIkairosfocus

March 16, 2007

March

03

Mar

16

16

2007

03:58 AM

3

03

58

AM

PDT

Copy Comment Link

GilDodgen: You are right, and as your linked review notes:

What happens when an immature and incomplete science meets a societal demand for information and direction? The spectacle is not pretty, as we learn from Useless Arithmetic, a new book that describes a long list of incompetent and sometimes mindless uses of fragmentary scientific ideas in the realm of public policy. The troubling anecdotes that authors Orrin H. Pilkey and Linda Pilkey-Jarvis provide cross diverse fields, including fisheries management, nuclear-waste disposal, beach erosion, climate change, ore mining, seed dispersal and disease control. Their extended examples of the misuse of science are both convincing and depressing. The book is a welcome antidote to the blind use of supposedly quantitative models, which may well represent the best one can do, but which are not yet capable of producing useful information . . .

The book then goes on to try to say the problem is that his is not science but engineering, the application of science to practical situations. First, that is probably not an historically justified "definition" of either science or engineering. However, he is right once he gets into:

politics, economics, the legal system and even psychology are involved. When science is not ready to answer specific questions, but the political universe insists that policies must be put in place (How large a catch can the fishery sustain? Is malaria in Africa a greater problem than HIV? How rapidly will this beach erode?), the outcome is almost inevitable: Someone will rush forward claiming that the answer is at hand, and the political system, driven to cope with a public threat or desire, will likely implement some insupportable policy. When the science is incomplete, one enters the world of P. T. Barnum, medical nostrums and the carnival.

I would add to that the underlying issue at stake: PHILOSOPHY. Specifically, many in the west have a worldview in which Science is the substitute for prophets, and so many are tempted to the mantle of prophet-hood. Absent an all-knowing God to give you the prophetic word, inaccuracies, often material ones, are bound to creep in. But at the same time, there are serious epistemological [phil of knowledge] limitations to scientific claims, such that we should recognise that science is probably best viewed as . . .

provisional knowledge of the world based on observation, theoretical analysis and modelling, empirical testing, and debate among the community of the learned, within the wider context of the philosophy of knowledge

For instance, we often see an argument that runs in effect that if theory, then observations. Observations so theory. This affirms the consequent in logic: “If Tom is a cat then Tom is an animal” does not lead to “since Tom is an animal he must be a cat.” This fallacy is the root of the point that such theories are provisional at best. Thus we see Newtonian science triumphant for 200 years tot he 1880's then collapsing into a corner as a limiting case for slow, big bodies, over the next fifty years. In short, it is not just an issue of mere degree of maturity as a science – there is a fundamental limitation on knowledge of the empirical world. But, pride goeth before a fall, and is unwilling to learn such humbling lessons. Mix in agendas and politics, interest groups, dirty debate tactics etc, and the mix is explosive. Then, as you speak on, bring on the latest magic mirror on the wall: the computer, and ask it what the future holds, or what the past held. No computer simulation is better than its input data, assumptions and algorithms, not to mention coding. In the case of alleged evolutionary mechanisms, the past is inherently beyond reach so we are looking at plausibilities not knowledge, at best. Not to mention, targetted searches of short random text strings to get to methinks it is a weasel or whatever are instances of intelligent design, but a bit of distraction can transmute that rhetorically into evidence for Darwinian macroevolution. Those who want that to be the past will lap it up, and most do not know enough to spot the problem, maybe including the programmers in question. Similarly, the future decades or centuries hence is equally un-reacheable to our experiences in the present -- absent time travel -- and so we are locked up to uncertainties and the issues of prudence. We cannot reliably predict the weather one month from today, but claim to project the climate decades or centuries hence? Worse, with coarse grained, sometimes questionable data sets to calibrate the models? And with major unanswered questions on mechanisms, feedback loops and the directions of cause and effect? Then, to prevent the imagined climate catastrophe we set out to collapse energy use by 30 or more percent in a context where that use is directly connected to the scale of economic activity and a drop of a few percent in economic activity produces serious hardships? And in a world in which the last major global depression in part led to global war? I think the time has come for a sober assessment of what we know, what we do not know, and what our ways of deciding and acting may do to us, here and in the hereafter. [E.g. Do we really have good evidence and reason to redefine science in such a way as to leave the impression that there is no Creator and Lord to account to, especially in school systems and the public media?] Thanks for a thoughtful post GEM of TKIkairosfocus

March 16, 2007

March

03

Mar

16

16

2007

03:49 AM

3

03

49

AM

PDT

Copy Comment Link