Uncommon Descent Serving The Intelligent Design Community

# A Designed Object’s Entropy Must Increase for Its Design Complexity to Increase – Part 1

Share
Flipboard
Print
Email

The common belief is that adding disorder to a designed object will destroy the design (like a tornado passing through a city, to paraphrase Hoyle). Now if increasing entropy implies increasing disorder, creationists will often reason that “increasing entropy of an object will tend to destroy its design”. This essay will argue mathematically that this popular notion among creationists is wrong.

The correct conception of these matters is far more nuanced and almost the opposite of (but not quite) what many creationists and IDists believe. Here is the more correct view of entropy’s relation to design (be it man-made or otherwise):

1. increasing entropy can increase the capacity for disorder, but it doesn’t necessitate disorder

2. increasing an object’s capacity for disorder doesn’t imply that the object will immediately become more disordered

3. increasing entropy in a physical object is a necessary (but not sufficient) condition for increasing the complexity of the design

4. contrary to popular belief, a complex design is a high entropy design, not a low entropy design. The complex organization of a complex design is made possible (and simultaneously improbable) by the high entropy the object contains.

5. without entropy there is no design

If there is one key point it is: Entropy makes design possible but simultaneously improbable. And that is the nuance that many on both sides of the ID/Creation/Evolution controversy seem to miss.

The notion of entropy is foundational to physics, engineering, information theory and ID. These essays are written to provide a discussion on the topic of entropy and its relationship to other concepts such as uncertainty, probability, microstates, and disorder. Much of what is said will go against popular understanding, but the aim is to make these topics clearer. Some of the math will be in a substantially simplified form, so apologies in advance to the formalists out there.

Entropy may refer to:

1. Thermodynamic (Statistical Mechanics) entropy – measured in Joules/Kelvin, dimensionless units, degrees of freedom, or (if need be) bits

2. Shannon entropy – measured in bits or dimensionless units

3. Algorithmic entropy or Kolmogorov complexity – measured also in bits, but deals with the compactness of a representation. A file that can be compressed substantially has low algorithmic entropy, whereas files which can’t be compressed evidence high algorithmic entropy (Kolmogorov complexity). Both Shannon entropy and algorithmic entropies are within the realm of information theory, but by default, unless otherwise stated, most people associate Shannon entropy as the entropy in information theory.

4. disorder in the popular sense – no real units assigned, often not precise enough to be of scientific or engineering use. I’ll argue the term “disorder” is a misleading way to conceptualize entropy. Unfortunately, the word “disorder” is used even in university science books. I will argue mathematically why this is so…

The reason the word entropy is used in the disciplines of Thermodynamics, Statistical Mechanics and Information Theory is that there are strong mathematical analogies. The evolution of the notion of entropy began with Clausius who also coined the term for thermodynamics, then Boltzmann and Gibbs related Clausius’s notions of entropy to Newtonian (Classical) Mechanics, then Shannon took Boltzmann’s math and adapted it to information theory, and then Landauer brought things back full circle by tying thermodynamics to information theory.

How entropy became equated with disorder, I do not know, but the purpose of these essays is to walk through actual calculations of entropy and allow the reader to decide for himself whether disorder can be equated with entropy. My personal view is that Shannon entropy and Thermodynamic entropy cannot be equated with disorder, even though the lesser-known algorithmic entropy can. So in general entropy should not be equated with disorder. Further, the problem of organization (which goes beyond simple notions of order and entropy) needs a little more exploration. Organization sort of stands out as a quality that seems difficult to assign numbers to.

The calculations that follow are to give an illustration how I arrived at some my conclusions.

First I begin with calculating Shannon entropy for simple cases. Thermodynamic entropy will be covered in the Part II.

Bill Dembski actually alludes to Shannon entropy in his latest offering on Conservation of Information Made Simple

In the information-theory literature, information is usually characterized as the negative logarithm to the base two of a probability (or some logarithmic average of probabilities, often referred to as entropy).

William Dembski
Conservation of Information Made Simple

To elaborate on what Bill said, if we have a fair coin, it can exist in two microstates: heads (call it microstate 1) or tails (call it microstate 2).

After a coin flip, the probability of the coin emerging in microstate 1 (heads) is 1/2. Similarly the probability of the coin emerging in microstate 2 (tails) is 1/2. So let me tediously summarize the facts:

N = Ω(N) = Ω = Number of microstates of a 1-coin system = 2

x1 = microstate 1 = heads
x2 = microstate 2 = tails

P(x1) = P(microstate 1)= P(heads) = probability of heads = 1/2
P(x2) = P(microstate 2)= P(tails) = probability of tails = 1/2

Here is the process for calculating the Shannon Entropy of a 1-coin information system starting with Shannon’s famous formula:

$\large I=-\sum_{i=0}^n {p({x}_{i})\log_{2}p(x_{i})}$

$\large =-p({x}_{1})\log_{2}p(x_{1})-p({x}_{2})\log_{2}p(x_{2})$

$\large =-p(\text{heads})\log_{2}p(\text{heads})-p(\text{tails})\log_{2}p(\text{tails})$

$\large =-(\frac{1}{2})\log_{2}(\frac{1}{2})-(\frac{1}{2})\log_{2}(\frac{1}{2})$

$\large =\frac{1}{2}+\frac{1}{2}= 1 = 1 \text { bit}$

where I is the Shannon entropy (or measure of information).

This method seems a rather torturous way to calculate the Shannon entropy of a single coin. A slightly simpler method exists if we take advantage of the fact that each microstate of the coin (heads or tails) is equiprobable, and thus conforms to the fundamental postulate of statistical mechanics, and thus we can calculate the number of bits by simply taking the logarithm of the number of microstates as is done in statistical mechanics.

$\large I=-\sum_{i=0}^n {p({x}_{i})\log_{2}p(x_{i})}$

$\large =\log_{2}\Omega =\log_{2}(2)=1=1 \text{ bit}$

Now compare this equation of the Shannon entropy in information theory

$\large I=\log_{2}\Omega$

to Boltzmann entropy from statistical mechanics and thermodynamics

$\large S=k_{b}\ln\Omega$

and even more so using different units whereby kb=1

$\large S=\ln\Omega$

The similarities are not an accident. Shannon’s ideas of information theory are a descendant of Boltzmann’s ideas from statistical mechanics and thermodynamics.

To explore Shannon entropy further, let us suppose we have a system of 3 distinct coins. The Shannon entropy relates the amount of information that will be gained by observing the collective state (microstate) of the 3 coins.

First we have to compute the number of microstates or ways the system of coins can be configured. I will lay them out specifically.

microstate 1 = H H H
microstate 2 = H H T
microstate 3 = H T H
microstate 4 = H T T
microstate 5 = T H H
microstate 6 = T H T
microstate 7 = T T H
microstate 8 = T T T

N = Ω(N) = Ω = Number of microstates of a 3-coin system = 8

So there are 8 microstates or outcomes the system can realize. The Shannon entropy can be calculated in the torturous way:
$\small I=-\sum_{i=0}^n {p({x}_{i})log_{2}p(x_{i})}$

$=-p(\text{hhh})\log_{2}p(\text{hhh}) -p(\text{hht})\log_{2}p(\text{hht}) -p(\text{hth})\log_{2}p(\text{hth}) -p(\text{htt})\log_{2}p(\text{htt}) -p(\text{thh})\log_{2}p(\text{thh}) -p(\text{tht})\log_{2}p(\text{tht}) -p(\text{tth})\log_{2}p(\text{tth}) -p(\text{ttt})\log_{2}p(\text{ttt})$

$= -\frac{1}{8}\log_{2}(\frac{1}{8}) -\frac{1}{8}\log_{2}(\frac{1}{8}) -\frac{1}{8}\log_{2}(\frac{1}{8}) -\frac{1}{8}\log_{2}(\frac{1}{8}) -\frac{1}{8}\log_{2}(\frac{1}{8}) -\frac{1}{8}\log_{2}(\frac{1}{8}) -\frac{1}{8}\log_{2}(\frac{1}{8}) -\frac{1}{8}\log_{2}(\frac{1}{8})$

$=3=3 \text{ bits}$

or simply taking the logarithm of the number of microstates:

$I=\log_{2}\Omega =\log_{2}8=3=3 \text{ bits}$

It can be shown that for the Shannon entropy of a system of N distinct coins is equal to N bits. That is, a system with 1 coin has 1 bit of Shannon entropy, a system with 2 coins has 2 bits of Shannon entropy, a system of 3 coins has 3 bits of Shannon entropy, etc.

Notice, the more microstates there are, the more uncertainty exists that the system will be found in any given microstate. Equivalently, the more microstates there are, the more improbable the system will be found in a given microstate. Hence, sometimes entropy is described in terms of improbability or uncertainty or unpredictability. But we must be careful here, uncertainty is not the same thing as disorder. That is subtle but important distinction.

So what is the Shannon Entropy of a system of 500 distinct coins? Answer: 500 bits, or the Universal Probability Bound.

By way of extension, if we wanted to build an operating system like Windows-7 that requires gigabits of storage, we would require the computer memory to contain gigabits of Shannon entropy. This illustrates the principle that more complex designs require larger Shannon entropy to support the design. It cannot be otherwise. Design requires the presence of entropy, not absence of it.

Suppose we found that a system of 500 coins were all heads, what is the Shannon entropy of this 500-coin system? Answer: 500 bits. No matter what configuration the system is in, whether ordered (like all heads) or disordered, the Shannon entropy remains the same.

Now suppose a small tornado went through the room where the 500 coins resided (with all heads before the tornado), what is the Shannon entropy after the tornado? Same as before, 500-bits! What may arguably change is the algorithmic entropy (Kolmogorov complexity). The algorithmic entropy may go up, which simply means we can’t represent the configuration of the coins in a compact sort of way like saying “all heads” or in the Kleene notation as H*.

Amusingly, if in the aftermath of the tornado’s rampage, the room got cooler, the thermodynamic entropy of the coins would actually go down! Hence the order or disorder of the coins is independent not only of the Shannon entropy but also the thermodynamic entropy.

Let me summarize the before and after of the tornado going through the room with the 500 coins:

BEFORE : 500 coins all heads, Temperature 80 degrees
Shannon Entropy : 500 bits
Algorithmic Entropy (Kolmogorov complexity): low
Thermodynamic Entropy : some finite starting value

AFTER : 500 coins disordered
Shannon Entropy : 500 bits
Algorithmic Entropy (Kolmogorov complexity): high
Thermodynamic Entropy : lower if the temperature is lower, higher if the temperature is higher

Now to help disentangle concepts a little further consider three 3 computer files:

File_A : 1 gigabit of binary numbers randomly generated
File_B : 1 gigabit of all 1’s
File_C : 1 gigabit encrypted JPEG

Here are the characteristics of each file:

File_A : 1 gigabit of binary numbers randomly generated
Shannon Entropy: 1 gigabit
Algorithmic Entropy (Kolmogorov Complexity): high
Thermodynamic Entropy: N/A
Organizational characteristics: highly disorganized
inference : not designed

File_B : 1 gigabit of all 1’s
Shannon Entropy: 1 gigabit
Algorithmic Entropy (Kolmogorov Complexity): low
Thermodynamic Entropy: N/A
Organizational characteristics: highly organized
inference : designed (with qualification, see note below)

File_C : 1 gigabit encrypted JPEG
Shannon Entropy: 1 gigabit
Algorithmic Entropy (Kolmogorov complexity): high
Thermodynamic Entropy: N/A
Organizational characteristics: highly organized
inference : extremely designed

Notice, one cannot ascribe high levels of improbable design based on the Shannon entropy or algorithmic entropy without some qualification. Existence of improbable design depends on the existence of high Shannon entropy, but is somewhat independent of algorithmic entropy. Further, to my knowledge, there is not really a metric for organization that is separate from Kolmogorov complexity, but this definition needs a little more exploration and is beyond my knowledge base.

Only in rare cases will high Shannon entropy and low algorithmic entropy (Kolmogorov complexity) result in a design inference. One such example is 500 coins all heads. The general method to infer design (including man-made designs), is that the object:

1. has High Shannon Entropy (high improbability)
2. conforms to an independent (non-postdictive) specification

In contrast to the design of coins being all heads where the Shannon entropy is high but the algorithmic entropy is low, in cases like software or encrypted JPEG files, the design exists in an object that has both high Shannon entropy and high algorithmic entropy. Hence, the issues of entropy are surely nuanced, but on balance entropy is good for design, not always bad for it. In fact, if an object evidences low Shannon entropy, we will not be able to infer design reliably.

The reader might be disturbed at my final conclusion in as much as it grates against popular notions of entropy and creationist notions of entropy. But well, I’m no stranger to this controversy. I explored Shannon entropy in this thread because it is conceptually easier than its ancestor concept of thermodynamic entropy.

In the Part II (which will take a long time to write) I’ll explore thermodynamic entropy and its relationship (or lack thereof) to intelligent design. But in brief, a parallel situation often arises: the more complex a design, the higher its thermodynamic entropy. Why? The simple reason is that more complex designs involve more parts (molecules) and more molecules in general imply higher thermodynamic (as well as Shannon) entropy. So the question of Earth being an open system is a bit beside the point since entropy is essential for intelligent designs to exist in the first place.

[UPDATE: the sequel to this thread is in Part 2]

Acknowledgements (both supporters and critics):

1. Elizabeth Liddle for hosting my discussions on the 2nd Law at TheSkepticalZone

2. physicist Olegt who offered generous amounts of time in plugging the holes in my knowledge, particularly regarding the Liouville Theorem and Configurational Entropy

3. retired physicist Mike Elzinga for his pedagogical examples and historic anecdotes. HT: the relationship of more weight to more entropy

4. An un-named theoretical physicist who spent many hours teaching his students the principles of Statistical Mechanics and Thermodynamics

5. physicists Andy Jones and Rob Sheldon

6. Neil Rickert for helping me with Latex

7. Several others that have gone unnamed

NOTE:
[UPDATE and correction: gpuccio was kind enough to point out that in the case of File_B, the design inference isn’t necessarily warranted. It’s possible an accident or programming error or some other reason could make all the bits 1. It would only be designed if that was the designer’s intention.]

[UPDATE 9/7/2012]
Boltzmann

“In order to explain the fact that the calculations based on this assumption [“…that by far the largest number of possible states have the characteristic properties of the Maxwell distribution…”] correspond to actually observable processes, one must assume that an enormously complicated mechanical system represents a good picture of the world, and that all or at least most of the parts of it surrounding us are initially in a very ordered — and therefore very improbable — state. When this is the case, then whenever two of more small parts of it come into interaction with each other, the system formed by these parts is also initially in an ordered state and when left to itself it rapidly proceeds to the disordered most probable state.” (Final paragraph of #87, p. 443.)

That slight, innocent paragraph of a sincere man — but before modern understanding of q(rev)/T via knowledge of molecular behavior (Boltzmann believed that molecules perhaps could occupy only an infinitesimal volume of space), or quantum mechanics, or the Third Law — that paragraph and its similar nearby words are the foundation of all dependence on “entropy is a measure of disorder”. Because of it, uncountable thousands of scientists and non-scientists have spent endless hours in thought and argument involving ‘disorder’and entropy in the past century. Apparently never having read its astonishingly overly-simplistic basis, they believed that somewhere there was some profound base. Somewhere. There isn’t. Boltzmann was the source and no one bothered to challenge him. Why should they?

Boltzmann’s concept of entropy change was accepted for a century primarily because skilled physicists and thermodynamicists focused on the fascinating relationships and powerful theoretical and practical conclusions arising from entropy’s relation to the behavior of matter. They were not concerned with conceptual, non-mathematical answers to the question, “What is entropy, really?” that their students occasionally had the courage to ask. Their response, because it was what had been taught to them, was “Learn how to calculate changes in entropy. Then you will understand what entropy ‘really is’.”

There is no basis in physical science for interpreting entropy change as involving order and disorder.

PS: The most probable or equilibrium cluster of microstates consistent with a given macrostate, is the cluster that has the least information about it, and the most freedom of variation of mass and energy distribution at micro level. This high entropy state-cluster is strongly correlated with high levels of disorder, for reasons connected to the functionality constraints just above. And in fact -- never mind those who are objecting and pretending that this is not so -- it is widely known in physics that entropy is a metric of disorder, some would say it quantifies it and gives it structured physical expression in light of energy and randomness or information gap considerations. kairosfocus
Mung: One more time [cf. 56 above, which clips elsewhere . . . ], let me clip Shannon, 1950/1:
The entropy is a statistical parameter which measures, in a certain sense, how much information is produced on the average for each letter of a text in the language. If the language is translated into binary digits (0 or 1) in the most efficient way, the entropy is the average number of binary digits required per letter of the original language. The redundancy, on the other hand, measures the amount of constraint imposed on a text in the language due to its statistical structure, e.g., in English the high fre-quency of the letter E, the strong tendency of H to follow T or of V to follow Q. It was estimated that when statistical effects extending over not more than eight letters are considered the entropy is roughly 2.3 bits per letter, the redundancy about 50 per cent.
Going back to my longstanding, always linked note, which I have clipped several times over the past few days, here on is how we measure info and avg info per symbol:
What this last refers to is the Gibbs formulation of entropy for statistical mechanics, and its implications when the relationship between probability and information is brought to bear in light of the Macro-micro views of a body of matter. That is, when we have a body, we can characterise its state per lab-level thermodynamically significant variables, that are reflective of many possible ultramicroscopic states of constituent particles. Thus, clipping again from my always linked discussion that uses Robertson's Statistical Thermophysics, CH 1 [and do recall my strong recommendation that we all acquire and read L K Nash's elements of Statistical Thermodynamics as introductory reading):
Summarising Harry Robertson's Statistical Thermophysics (Prentice-Hall International, 1993) . . . . For, as he astutely observes on pp. vii - viii:
. . . the standard assertion that molecular chaos exists is nothing more than a poorly disguised admission of ignorance, or lack of detailed information about the dynamic state of a system . . . . If I am able to perceive order, I may be able to use it to extract work from the system, but if I am unaware of internal correlations, I cannot use them for macroscopic dynamical purposes. On this basis, I shall distinguish heat from work, and thermal energy from other forms . . .
And, in more details, (pp. 3 - 6, 7, 36, cf Appendix 1 below for a more detailed development of thermodynamics issues and their tie-in with the inference to design; also see recent ArXiv papers by Duncan and Samura here and here):
. . . It has long been recognized that the assignment of probabilities to a set represents information, and that some probability sets represent more information than others . . . if one of the probabilities say p2 is unity and therefore the others are zero, then we know that the outcome of the experiment . . . will give [event] y2. Thus we have complete information . . . if we have no basis . . . for believing that event yi is more or less likely than any other [we] have the least possible information about the outcome of the experiment . . . . A remarkably simple and clear analysis by Shannon [1948] has provided us with a quantitative measure of the uncertainty, or missing pertinent information, inherent in a set of probabilities [NB: i.e. a probability different from 1 or 0 should be seen as, in part, an index of ignorance] . . . . [deriving informational entropy, cf. discussions here, here, here, here and here; also Sarfati's discussion of debates and the issue of open systems here . . . ] H({pi}) = - C [SUM over i] pi*ln pi, [. . . "my" Eqn 6] [--> This is essentially the same as Gibbs Entropy, once C is properly interpreted and the pi's relate to the probabilities of microstates consistent with the given lab-observable macrostate of a system at a given Temp, with a volume V, under pressure P, degree of magnetisation, etc etc . . . ] [where [SUM over i] pi = 1, and we can define also parameters alpha and beta such that: (1) pi = e^-[alpha + beta*yi]; (2) exp [alpha] = [SUM over i](exp - beta*yi) = Z [Z being in effect the partition function across microstates, the "Holy Grail" of statistical thermodynamics]. . . . [H], called the information entropy, . . . correspond[s] to the thermodynamic entropy [i.e. s, where also it was shown by Boltzmann that s = k ln w], with C = k, the Boltzmann constant, and yi an energy level, usually ei, while [BETA] becomes 1/kT, with T the thermodynamic temperature . . . A thermodynamic system is characterized by a microscopic structure that is not observed in detail . . . We attempt to develop a theoretical description of the macroscopic properties in terms of its underlying microscopic properties, which are not precisely known. We attempt to assign probabilities to the various microscopic states . . . based on a few . . . macroscopic observations that can be related to averages of microscopic parameters. Evidently the problem that we attempt to solve in statistical thermophysics is exactly the one just treated in terms of information theory. It should not be surprising, then, that the uncertainty of information theory becomes a thermodynamic variable when used in proper context . . . . Jayne's [summary rebuttal to a typical objection] is ". . . The entropy of a thermodynamic system is a measure of the degree of ignorance of a person whose sole knowledge about its microstate consists of the values of the macroscopic quantities . . . which define its thermodynamic state. This is a perfectly 'objective' quantity . . . it is a function of [those variables] and does not depend on anybody's personality. There is no reason why it cannot be measured in the laboratory." . . . . [pp. 3 - 6, 7, 36; replacing Robertson's use of S for Informational Entropy with the more standard H.]
As is discussed briefly in Appendix 1, Thaxton, Bradley and Olsen [TBO], following Brillouin et al, in the 1984 foundational work for the modern Design Theory, The Mystery of Life's Origins [TMLO], exploit this information-entropy link, through the idea of moving from a random to a known microscopic configuration in the creation of the bio-functional polymers of life, and then -- again following Brillouin -- identify a quantitative information metric for the information of polymer molecules. For, in moving from a random to a functional molecule, we have in effect an objective, observable increment in information about the molecule. This leads to energy constraints, thence to a calculable concentration of such molecules in suggested, generously "plausible" primordial "soups." In effect, so unfavourable is the resulting thermodynamic balance, that the concentrations of the individual functional molecules in such a prebiotic soup are arguably so small as to be negligibly different from zero on a planet-wide scale. By many orders of magnitude, we don't get to even one molecule each of the required polymers per planet, much less bringing them together in the required proximity for them to work together as the molecular machinery of life. The linked chapter gives the details. More modern analyses [e.g. Trevors and Abel, here and here], however, tend to speak directly in terms of information and probabilities rather than the more arcane world of classical and statistical thermodynamics . . .
Now, of course, as Wiki summarises, the classic formulation of the Gibbs entropy is:
The macroscopic state of the system is defined by a distribution on the microstates that are accessible to a system in the course of its thermal fluctuations. So the entropy is defined over two different levels of description of the given system. The entropy is given by the Gibbs entropy formula, named after J. Willard Gibbs. For a classical system (i.e., a collection of classical particles) with a discrete set of microstates, if E_i is the energy of microstate i, and p_i is its probability that it occurs during the system's fluctuations, then the entropy of the system is: S = -k_B * [sum_i] p_i * ln p_i This definition remains valid even when the system is far away from equilibrium. Other definitions assume that the system is in thermal equilibrium, either as an isolated system, or as a system in exchange with its surroundings. The set of microstates on which the sum is to be done is called a statistical ensemble. Each statistical ensemble (micro-canonical, canonical, grand-canonical, etc.) describes a different configuration of the system's exchanges with the outside, from an isolated system to a system that can exchange one more quantity with a reservoir, like energy, volume or molecules. In every ensemble, the equilibrium configuration of the system is dictated by the maximization of the entropy of the union of the system and its reservoir, according to the second law of thermodynamics (see the statistical mechanics article). Neglecting correlations between the different possible states (or, more generally, neglecting statistical dependencies between states) will lead to an overestimate of the entropy[1]. These correlations occur in systems of interacting particles, that is, in all systems more complex than an ideal gas. This S is almost universally called simply the entropy. It can also be called the statistical entropy or the thermodynamic entropy without changing the meaning. Note the above expression of the statistical entropy is a discretized version of Shannon entropy. The von Neumann entropy formula is an extension of the Gibbs entropy formula to the quantum mechanical case. It has been shown that the Gibb's Entropy is numerically equal to the experimental entropy[2] dS = delta_Q/{T} . . .
Looks to me that this is one time Wiki has it just about dead right. Let's deduce a relationship that shows physical meaning in info terms, where (- log p_i) is an info metric, I-i, here for microstate i, and noting that a sum over i of p_i * log p_i is in effect a frequency/probability weighted average or the expected value of the log p_i expression, and also moving away from natural logs (ln) to generic logs:
S_Gibbs = -k_B * [sum_i] p_i * log p_i But, I_i = - log p_i So, S_Gibbs = k_B * [sum_i] p_i * I-i i.e. S-Gibbs is a constant times the average information required to specify the particular microstate of the system, given its macrostate, the MmIG (macro-micro info gap.
Or, as Wiki also says elsewhere:
At an everyday practical level the links between information entropy and thermodynamic entropy are not close. Physicists and chemists are apt to be more interested in changes in entropy as a system spontaneously evolves away from its initial conditions, in accordance with the second law of thermodynamics, rather than an unchanging probability distribution. And, as the numerical smallness of Boltzmann's constant kB indicates, the changes in S / kB for even minute amounts of substances in chemical and physical processes represent amounts of entropy which are so large as to be right off the scale compared to anything seen in data compression or signal processing. But, at a multidisciplinary level, connections can be made between thermodynamic and informational entropy, although it took many years in the development of the theories of statistical mechanics and information theory to make the relationship fully apparent. In fact, in the view of Jaynes (1957), thermodynamics should be seen as an application of Shannon's information theory: the thermodynamic entropy is interpreted as being an estimate of the amount of further Shannon information needed to define the detailed microscopic state of the system, that remains uncommunicated by a description solely in terms of the macroscopic variables of classical thermodynamics. For example, adding heat to a system increases its thermodynamic entropy because it increases the number of possible microscopic states that it could be in, thus making any complete state description longer. (See article: maximum entropy thermodynamics.[Also,another article remarks: >>in the words of G. N. Lewis writing about chemical entropy in 1930, "Gain in entropy always means loss of information, and nothing more" . . . in the discrete case using base two logarithms, the reduced Gibbs entropy is equal to the minimum number of yes/no questions that need to be answered in order to fully specify the microstate, given that we know the macrostate.>>]) Maxwell's demon can (hypothetically) reduce the thermodynamic entropy of a system by using information about the states of individual molecules; but, as Landauer (from 1961) and co-workers have shown, to function the demon himself must increase thermodynamic entropy in the process, by at least the amount of Shannon information he proposes to first acquire and store; and so the total entropy does not decrease (which resolves the paradox).
So, immediately, the use of "entropy" in the Shannon context, to denote not H but N*H, where N is the number of symbols (thus, step by step states emitting those N symbols involved), is an error of loose reference. Similarly, by exploiting parallels in formulation and insights into the macro-micro distinction in thermodynamics, we can develop a reasonable and empirically supportable physical account of how Shannon information is a component of the Gibbs entropy narrative. Where also Gibbs subsumes the Boltzmann formulation and onward links to the lab-measurable quantity. (Nash has a useful, relatively lucid -- none of this topic is straightforward -- discussion on that.) Going beyond, once the bridge is there between information and entropy, it is there. It is not going away, regardless of how inconvenient it may be to some schools of thought. We can easily see that, for example, information is expressed in the configuration of a string, Z, of elements z1 -z2 . . . zN in accordance with a given protocol of assignment rules and interpretation & action rules etc. Where also, such is WLOG as AutoCAD etc show us that using the nodes and arcs representation and a list of structured strings that record this, essentially any object can be described in terms of a suitably configured string or collection of strings. So now, we can see that string Z (with each zi possibly taking b discrete states) may represent an island of function that expresses functionally specific complex organisation and associated information. Because of specificity to achieve and keep function, leading to a demand for matching, co-ordinated values of zi along the string, that string has relatively few of the N^b possibilities for N elements with b possible states being permissible. We are at isolated islands of specific function i.e cases E from a zone of function T in a space of possibilities W. (BTW, once b^N exceeds 500 bits on the gamut of our solar system, or 1,000 bits on the gamut of our observable cosmos, that brings to bear all the needle in the haystack, monkeys at keyboards analysis that has been repeatedly brought forth to show why FSCO/I is a useful sign of IDOW -- intelligently directed organising work -- as empirically credible cause.) We see then that we have a complex string to deal with, with sharp restrictions on possible configs, that are evident from observable function, relative to the general possibility of W = b^N possibilities. Z is in a highly informational, tightly constrained state that comes form a special zone specifiable on macro-level observable function (without actually observing Z directly). That constraint on degrees of freedom contingent on functional, complex organisation, is tantamount to saying that a highly informational state is a low entropy one, in the Gibbs sense. Going back to the expression, comparatively speaking there is not a lot of MISSING micro-level info to be specified, i.e. simply by knowing the fact of complex specified information-rich function, we know that we are in a highly restricted special Zone T in W. This immediately applies to R/DNA and proteins, which of course use string structures. It also applies tot he complex 3-D arrangement of components in the cell, which are organised in ways that foster function. And of course it applies to the 747 in a flyable condition. Such easily explains why a tornado passing through a junkyard in Seattle will not credibly assemble a 747 from parts it hits, and it explains why the raw energy and forces of the tornado that hits another formerly flyable 747, and tearing it apart, would render its resulting condition much less specified per function, and in fact result in predictable loss of function. We will also see that this analysis assumes the functional possibilities of a mass of Al, but is focussed on the issue of functional config and gives it specific thermodynamics and information theory context. (Where also, algebraic modelling is a valid mathematical analysis.) I trust this proves helpful KF kairosfocus
They were not concerned with conceptual, non-mathematical answers to the question, “What is entropy, really?” that their students occasionally had the courage to ask.
Does Lambert answer that question? What is Entropy, really? So, entropy is the answer to the age-old question, why me? Mung
...and when left to itself it rapidly proceeds to the most probable state.
There, I fixed it fer ya! As a bonus you get the "directionality" of entropy. Ordered and disordered gots nothing to do with it. Mung
Boltzmann
“In order to explain the fact that the calculations based on this assumption [“…that by far the largest number of possible states have the characteristic properties of the Maxwell distribution…”] correspond to actually observable processes, one must assume that an enormously complicated mechanical system represents a good picture of the world, and that all or at least most of the parts of it surrounding us are initially in a very ordered — and therefore very improbable — state. When this is the case, then whenever two of more small parts of it come into interaction with each other, the system formed by these parts is also initially in an ordered state and when left to itself it rapidly proceeds to the disordered most probable state.” (Final paragraph of #87, p. 443.)
That slight, innocent paragraph of a sincere man — but before modern understanding of q(rev)/T via knowledge of molecular behavior (Boltzmann believed that molecules perhaps could occupy only an infinitesimal volume of space), or quantum mechanics, or the Third Law — that paragraph and its similar nearby words are the foundation of all dependence on “entropy is a measure of disorder”. Because of it, uncountable thousands of scientists and non-scientists have spent endless hours in thought and argument involving ‘disorder’and entropy in the past century. Apparently never having read its astonishingly overly-simplistic basis, they believed that somewhere there was some profound base. Somewhere. There isn’t. Boltzmann was the source and no one bothered to challenge him. Why should they? Boltzmann’s concept of entropy change was accepted for a century primarily because skilled physicists and thermodynamicists focused on the fascinating relationships and powerful theoretical and practical conclusions arising from entropy’s relation to the behavior of matter. They were not concerned with conceptual, non-mathematical answers to the question, “What is entropy, really?” that their students occasionally had the courage to ask. Their response, because it was what had been taught to them, was “Learn how to calculate changes in entropy. Then you will understand what entropy ‘really is’.” There is no basis in physical science for interpreting entropy change as involving order and disorder.
scordova
From the OP:
How entropy became equated with disorder, I do not know ...
Arieh Ben-Naim writes: "It should be noted that Boltzmann himself was perhaps the first to use the "disorder" metaphor in his writing:
...are initially in a very ordered - therefore very improbable - state ... when left to itself it rapidly proceeds to the disordered most probable state. - Boltzmann (1964)
You should note that Boltzmann uses the terms "order" and "disorder" as qualitative descriptions of what goes on in the system. When he defines entropy, however, he uses either the number of states or probability. Indeed, there are many examples where the term "disorder" can be applied to describe entropy. For instance, mixing two gases is well described as a process leading to a higher degree of disorder. However, there are many examples for which the disorder metaphor fails." Mung
As I have said above, the adoption of the term "entropy" for SMI was an unfortunate event, not because entropy is not SMI, but because SMI is not entropy!.
*SMI - Shannon's Measure of Information http://www.worldscientific.com/worldscibooks/10.1142/7694 Mung
OOPS, 600 + Exa BYTES kairosfocus
PS: As I head out, I think an estimate of what it would take to describe the state of 1 cc of monoatomic ideal gas at 760 mm HG and 0 degrees C, i.e. 2.687 * 10^19 particles with 6 degrees of positional and momentum freedom would help us. Let us devote 32 bits -- 16 bits to get 4 hex sig figs, and a sign bit plus 15 bits for the binary exponent to each of the (x, y, z) and (P_x, P-y and P-z) co-ordinates in the phase space. We are talking about:
2.687 * 10^19 particles x 32 bits per degree of freedom x 6 degrees of freedom each _____________ 5.159 * 10^21 bits of info
That is, to describe the state of the system at a given instant, we would need 5.159 * 10^21 bits, or 644.9 * 10^18 bits. That is how many yes/no quest5ions, in teh correct order, would have to be amnswered and processed every clock tick we update. And with 10^-14 s as a reasonable chemical reaction rate, we are seeing a huge amount of required processing to keep track. As to how that would be done, that is anybody's guess. kairosfocus
F/N: This from OP needs comment:
what is the Shannon Entropy of a system of 500 distinct coins? Answer: 500 bits, or the Universal Probability Bound. By way of extension, if we wanted to build an operating system like Windows-7 that requires gigabits of storage, we would require the computer memory to contain gigabits of Shannon entropy. This illustrates the principle that more complex designs require larger Shannon entropy to support the design. It cannot be otherwise. Design requires the presence of entropy, not absence of it.
Actually, in basic info theory, H strictly is a measure of average info content per element in a system or symbol in a message. Hence its being estimated on a weighted average of information per relevant element. This, I illustrated earlier from a Shannon 1950/1 paper, in comment 15 in the part 2 thread:
The entropy is a statistical parameter which measures, in a certain sense, how much information is produced on the average for each letter of a text in the language. If the language is translated into binary digits (0 or 1) in the most efficient way, the entropy is the average number of binary digits required per letter of the original language. The redundancy, on the other hand, measures the amount of constraint imposed on a text in the language due to its statistical structure, e.g., in English the high fre-quency of the letter E, the strong tendency of H to follow T or of V to follow Q. It was estimated that when statistical effects extending over not more than eight letters are considered the entropy is roughly 2.3 bits per letter, the redundancy about 50 per cent.
So, we see the context of usage here. But what happens when you have a message of N elements? In the case of a system of complexity N elements, then the cumulative, Shannon metric based information -- notice how I am shifting terms to avoid ambiguity -- is, logically, H + H + . . . H N times over, or N * H. And, as was repeatedly highlighted, in the case of the entropy of systems that are in clusters of microstates consistent with a macrostate, the thermodynamic entropy is usefully measured by and understood on terms of the Macro-micro information gap (MmIG], not on a per state or per particle basis but a cumulative basis: we know macro quantities, not the specific position and momentum of each particle, from moment to moment, which given chaos theory we could not keep track of anyway. A useful estimate per the Gibbs weighed probability sum entropy metric -- which is where Shannon reputedly got the term he used from in the first place, on a suggestion from von Neumann -- is:
>>in the words of G. N. Lewis writing about chemical entropy in 1930, "Gain in entropy always means loss of information, and nothing more" . . . in the discrete case using base two logarithms, the reduced Gibbs entropy is equal to the minimum number of yes/no questions that need to be answered in order to fully specify the microstate, given that we know the macrostate. >>
Where, Wiki gives a useful summary:
The macroscopic state of the system is defined by a distribution on the microstates that are accessible to a system in the course of its thermal fluctuations. So the entropy is defined over two different levels of description of the given system. The entropy is given by the Gibbs entropy formula, named after J. Willard Gibbs. For a classical system (i.e., a collection of classical particles) with a discrete set of microstates, if E_i is the energy of microstate i [--> Notice, summation is going to be over MICROSTATES . . . ], and p_i is its probability that it occurs during the system's fluctuations, then the entropy of the system is S_sys = - k_B [SUM over i's] P_i log p_i
Also, {- log p_i} is an information metric, I_i, i.e the information we would learn on actually coming to know that the system is in microstate i. Thus, we are taking a scaled info metric on the probabilistically weighted summmation of info in each microstate. Let us adjust: S_sys = k_B [SUM over i's] p_i * I_i This is the weighted average info per possible microstate, scaled by k_B. (Which of course is where the Joules per Kelvin come from.) In effect the system is giving us a message, its macrostate, but that message is ambiguous over the specific microstate in it. After a bit of mathematical huffing and puffing, we are seeing that the entropy is linked to the average info per possible microstate. Where this is going is of course that when a system is in a state with many possible microstates, it has enormous freedom of being in possible configs, but if the macro signals lock us down to specific states in small clusters, we need to account for how it could be in such clusters, when under reasonable conditions and circumstances, it could be easily in states that are far less specific. In turn that raises issues over IDOW. Which then points onward to FSCO/I being a sign of intelligent design. KF kairosfocus
But what if you only have the encoded string to work upon, and the JPEG codec generates an apparently random string as output? How do you tell whether the output signal is truly random or that it contains a human-readable message encoded using some other protocol?
You can't tell if a string is truly the product of mindless purposeless forces (random is your word), so you have to be agnostic about that. So one must accept that one can make a false inference to randomness (such as when someone wants to be extremely stealthy and encrypt the data). If it parses with another codec that is avaiable to you, you have good reason to accept the file is designed. Beyond that, one might have other techniques such as those that team Norton Symantec used to determine that Stuxnet was the product of an incredible level of intelligent design: How Digital Detectives Deciphered Stuxnet
And that illustrates how a non-random string in a computer might be deduced as the product of some serious ID. scordova
But even if you didn’t know in advance, the fact that a JPEG decoder could produce a meaningful image proves only that the message was encoded using the JPEG protocol.
And JPEG encoders are intellignetly deisgned, so the files generated are still products of intelligent design.
A magnificent feat of inference.
Indeed.
It might be interesting if you could prove that the message originated from a non-human source
Humans can make JPEGs, so no need to invoke non-human sources. scordova
No timothya- I don't think so. It is obvious. And not one of those biologists can produce any evidence that demonstrates otherwise. Joe
Joe I am clear that you think so. You are in disagreement with almost every practising biologist in the world of science. But that is your choice. In the meantime, can we focus on Sal's proposal? timothya
timothya- there isn't any evidence that natural selection is non-random- just so that we are clear. Joe
Joe posted this:
And timothya- I am still waiting for evidence that natural selection is non-random….
As far as it matters, you have already had your answer in a different thread. This thread seems to be focussed on the "how to identify designedness", so perhaps we should stick to that subject. timothya
Joe posted this:
I’m saying that if you find a file on a compuetr then it a given some agency put it there.
Brilliant insight. Users of computers generate artefacts that are stored in a form determined by the operating system of the computer that they are using (in turn determined by the human designers of the operating system involved). I would be a little surprised if it proved to be otherwise. However, the reliable transformation of input data to stored data in computer storage doesn't help Sal with his problem of how to assign "designedness" to an arbitrary string of input data. He has to show that there is a reliable way to distinguish between a genuinely random string and a pseudorandom string that is hiding a human-readable message, when all he has to go on is the string itself, with no prior knowledge. If he has such a method, I would be fascinated to know what it is. timothya
And timothya- I am still waiting for evidence that natural selection is non-random.... Joe
I'm saying that if you find a file on a compuetr then it a given some agency put it there. Joe
Joe posted this:
I would bet that both strings are the product of agency involvement as blind and undirected processes cannot construct a file.
Forget the container and consider the thing contained (I mean, really, do I have to define every parameter of the discussion?). Scientists sensing signals from a pulsar store the results in a computer "file" via a series of truth-preserving transformations (light data to electronics to magnetic marks on a hard drive). Are you arguing that the stored data does not correlate reliably to the original sense data? timothya
Waiting for Sal's response, I noticed that he posted this:
The fact that I knew File C was a JPEG suggests that I had some advanced knowledge of the file being designed. And even if I didn’t know that in advance, the fact that it could be parsed and processed as a JPEG indicates that it is organized.
Exactly. You knew in advance that the file was JPEG-encoded. But even if you didn't know in advance, the fact that a JPEG decoder could produce a meaningful image proves only that the message was encoded using the JPEG protocol. A magnificent feat of inference. It might be interesting if you could prove that the message originated from a non-human source. Otherwise not. But what if you only have the encoded string to work upon, and the JPEG codec generates an apparently random string as output? How do you tell whether the output signal is truly random or that it contains a human-readable message encoded using some other protocol? If I understand your original post, you claim that design is detectable from the pattern of the encoded message, independent of its mode of encoding. timothya
If you have a means of distinguishing between File X, (which contains a genuine random strong), and File Y (which contains a pseudorandom random string encoding a human-readable sentence), then fill your boots and publish the method.
I would bet that both strings are the product of agency involvement as blind and undirected processes cannot construct a file. Joe
If you have a means of distinguishing between File X, (which contains a genuine random strong), and File Y (which contains a pseudorandom random string encoding a human-readable sentence), then fill your boots and publish the method. The sound you can hear is that of computer security specialists the world over shifting uncomfortably in their seats. Or perhaps of computer security specialists laughing their faces off. The point is this: if you want to infer "design" solely from the evidence (of the contents of the files, with no a priori knowledge of their provenance), then what is your method? timothya
If you didn’t know in advance what the origin of File A and File C were, then you would have no useful evidence from the contents of the two files to decide that one was “highly disorganised” and the other was “highly organised”.
The fact that I knew File C was a JPEG suggests that I had some advanced knowledge of the file being designed. And even if I didn't know that in advance, the fact that it could be parsed and processed as a JPEG indicates that it is organized. The fact that I specified in advance that FILE A was created by a random number generator ensures a high probability it will not be designed. File B had to be restated with qualification as gpuccio pointed out. The inference of design or lack thereof was based on advanced prior knowledge, not some explantory filter after the fact. scordova
Correction, I posted this:
Your argument appears to say that if a system transmits a constant signal, then it must be organised.
I meant to use the term from your post that the valid inference for File B was that the file contents were designed. Clearly a gigabit of "ones" is organised in the sense that it has an evident pattern. timothya
Sal posted this:
Now to help disentangle concepts a little further consider three 3 computer files:
File_A : 1 gigabit of binary numbers randomly generated File_B : 1 gigabit of all 1?s File_C : 1 gigabit encrypted JPEG Here are the characteristics of each file: File_A : 1 gigabit of binary numbers randomly generated Shannon Entropy: 1 gigabit Algorithmic Entropy (Kolmogorov Complexity): high Thermodynamic Entropy: N/A Organizational characteristics: highly disorganized inference : not designed File_B : 1 gigabit of all 1?s Shannon Entropy: 1 gigabit Algorithmic Entropy (Kolmogorov Complexity): low Thermodynamic Entropy: N/A Organizational characteristics: highly organized inference : designed (with qualification, see note below) File_C : 1 gigabit encrypted JPEG Shannon Entropy: 1 gigabit Algorithmic Entropy (Kolmogorov complexity): high Thermodynamic Entropy: N/A Organizational characteristics: highly organized inference : extremely designed
Please tell me that you are joking. If you didn't know in advance what the origin of File A and File C were, then you would have no useful evidence from the contents of the two files to decide that one was "highly disorganised" and the other was "highly organised". Hint: the purpose of encryption is to make the contents of the file approach as closely as possible to a randomly generated string. File B supports an inference of "highly organised"? How? Why? What if the ground state of the signal is just the continuous emission of something interpreted digitally as "ones" (or zeroes" for that matter). Your argument appears to say that if a system transmits a constant signal, then it must be organised. timothya
FROM MUNG: No Sal, 500 pennies gets you 500 bits of copper plated zinc, not 500 bits of information (or Shannon entropy).
Contrast to Bill Dembski's recent article:
FROM BILL DEBMSKI In the information-theory literature, information is usually characterized as the negative logarithm to the base two of a probability (or some logarithmic average of probabilities, often referred to as entropy). This has the effect of transforming probabilities into bits and of allowing them to be added (like money) rather than multiplied (like probabilities). Thus, a probability of one-eighths, which corresponds to tossing three heads in a row with a fair coin, corresponds to three bits,
I just did a comparable calculation more elaborately, and you missed it. Instead of tossing a single coin 3 times, I had 3 coins tossed 1 time.
FROM WIKI A single toss of a fair coin has an entropy of one bit. A series of two fair coin tosses has an entropy of two bits. The entropy rate for the coin is one bit per toss
I wrote the analogous situation, except insted of making multiple tosses of a single coin, I did the formula for single tosses of multiple coins. The Shannon entropy is analogous.
I wrote: It can be shown that for the Shannon entropy of a system of N distinct coins is equal to N bits. That is, a system with 1 coin has 1 bit of Shannon entropy, a system with 2 coins has 2 bits of Shannon entropy, a system of 3 coins has 3 bits of Shannon entropy, etc.
scordova
Entropy:
The notion of entropy is foundational to physics, engineering, information theory and ID. These essays are written to provide a discussion on the topic of entropy and its relationship to other concepts such as uncertainty, probability, microstates, and disorder. Much of what is said will go against popular understanding, but the aim is to make these topics clearer.
ok, so what is entropy?
First I begin with calculating Shannon entropy for simple cases.
ok, but first, what is "Shannon entropy"?
2. Shannon entropy – measured in bits or dimensionless units
Telling me it's measured in bits doesn't tell me what "it" is.
I is the Shannon entropy (or measure of information).
So "Shannon entropy" is a measure of information?
Hence, sometimes entropy is described in terms of improbability or uncertainty or unpredictability.
So Shannon entropy is a measure of what we don't know? More like a measure of non-information? Mung
To elaborate on what Bill said, if we have a fair coin, it can exist in two microstates: heads (call it microstate 1) or tails (call it microstate 2).
I have to disagree with Bill. I have a coin in my pocket and it's not in either the heads state or the tails state. Mung
Surely you mean “if we wanted to build an operating system like Windows-7 that requires gigabits of storage, we would require the computer memory to contain 32 bits or so of Shannon entropy”
Surely not. 32-bits (or 64 bits) refers to the number of bits available to address memeory, not the actual amount of memory Windows-7 requires. 32 bits can address 2^32 bytes of memory or 4 gigabytes directly. From the Windows website describing Vista (and the comment applies to other Windows operating systems)
One of the greatest advantages of using a 64-bit version of Windows Vista is the ability to access physical memory (RAM) that is above the 4-gigabyte (GB) range. This physical memory is not addressable by 32-bit versions of Windows Vista.
Windows x64 occupies about 16gigabytes. A byte being 8 bits implies 16 gigabytes is 16*8 = 128 gigabits. Thus the Shannon entropy required to represent windows-7 x64 is on the order of 128 gigabits. Shannon entropy is the amount of information that can be represented, not the number of bits required to locate an address in memory. scordova
"if we wanted to build an operating system like Windows-7 that requires gigabits of storage, we would require the computer memory to contain gigabits of Shannon entropy" Surely you mean "if we wanted to build an operating system like Windows-7 that requires gigabits of storage, we would require the computer memory to contain 32 bits or so of Shannon entropy" EndoplasmicMessenger
EA: Notice, I consistently speak of sampling a distribution of possibilities in a config space, where the atomic resources of solar system or observed cosmos are such that only a very small fraction can be sampled. For 500 bits, we talk of a one straw size sample to a cubical haystack 1,000 LY on the side, about as thick as the galaxy. With all but certainty, a blind, chance and necessity sample will be dominated by the bulk of the distribution. In short, it is maximally implausible that special zones will be sampled. KF PS: Have I been sufficiently clear in underscoring that in stat thermo-d the relevant info metric associated with entropy is a measure of the missing info to specify micro state given macro state? kairosfocus
Their arguments I find worthwhile. I don’t have any new theories to offer. Such an endeavor would be over my head anyway. I know too little to make much of a contribution to the debate beyond what you have seen at places like UD. Besides, blogs aren’t really for doing science, laboratories and libraries are better places for that. The internet is just for fun…
Sorry, I have to bring it down a notch. Just something that has been on my mind a long time butifnot
Trevors and Abel point out the necessity of Shannon entropy (uncertainty) to store information for life to replicate. Hence, they recognize that a sufficient amount of Shannon entropy is needed for life:
Chance and Necessity do not explain the Origin of Life No natural mechanism of nature reducible to law canexplain the high information content of genomes. This is a mathematical truism, not a matter subject to over-turning by future empirical data. The cause-and-e?ect necessity described by natural law manifests a probability approaching 1.0. Shannon uncertainty is a probability function (-log2 p). When the probability of naturallaw events approaches 1.0, the Shannon uncertaintycontent becomes miniscule (-log2p = log2 1.0=0 uncertainty). There is simply not enough Shannon uncertainty in cause-and-e?ect determinism and its reductionistic laws to retain instructions for life.
scordova
Sal:
Besides, blogs aren’t really for doing science, laboratories and libraries are better places for that. The internet is just for fun…
Spoken like a true academic elitist! :) Eric Anderson
Hi butifnot, I don't believe that evolutionists have proven their case. There are fruitful ways to criticize OOL and Darwinism, I just think that creationists will hurt themselves using the 2nd Law and Entropy arguments (for the reasons outlined in these posts). They need to move on to arguments that are more solid. What is persuasive to me are the cases of evolutionsits leaving the Darwin camp or OOL camp: Micahel Denton Jerry Fodor Masimo Piantelli Jack Trevors Hubert Yockey Richard Sternberg Dean Kenyon James Shapiro etc. Their arguments I find worthwhile. I don't have any new theories to offer. Such an endeavor would be over my head anyway. I know too little to make much of a contribution to the debate beyond what you have seen at places like UD. Besides, blogs aren't really for doing science, laboratories and libraries are better places for that. The internet is just for fun... Sal scordova
Sal, the time is ripe for a bold new thermo-entropy synthesis! Practically the sum of human knowledge is available in an instant for free. A continuing and wider survey, far wide of materialists, is needed before this endeavor can (should) be launched to fruition. Comments on Shannon
Shannon’s concept of information is adequate to deal with the storage and transmission of data, but it fails when trying to understand the qualitative nature of information. Theorem 3: Since Shannon’s definition of information relates exclusively to the statistical relationship of chains of symbols and completely ignores their semantic aspect, this concept of information is wholly unsuitable for the evaluation of chains of symbols conveying a meaning. In order to be able adequately to evaluate information and its processing in different systems, both animate and inanimate, we need to widen the concept of information considerably beyond the bounds of Shannon’s theory. Figure 4 illustrates how information can be represented as well as the five levels that are necessary for understanding its qualitative nature. Level 1: statistics Shannon’s information theory is well suited to an understanding of the statistical aspect of information. This theory makes it possible to give a quantitative description of those characteristics of languages that are based intrinsically on frequencies. However, whether a chain of symbols has a meaning is not taken into consideration. Also, the question of grammatical correctness is completely excluded at this level. http://creation.com/information-science-and-biology
The distinction (good question) between data and information (and much else) must be addressed to get to thermo-design-info theory. butifnot
Part two is now available: Part II scordova
Sal, something's missing, don't you think? Does it not 'feel' that when we get to thermo and information and design, there is *more*, that will not to be admitted from a basic rehash, which is where it looks like you're at. The bridge between thermo and 'information' is fascinating, but here is where it could become really interesting - [what if] actual information has material and non material components! Our accounting may, and may have to, meet this reality. The difference in entropy of a 'live' brain and and the same brain dead with a small .22 hole in it is said to be very small, but is it? Perhaps something is missing. butifnot
OT:
Amazing --- light filmed at 1,000,000,000,000 Frames/Second! - video (this is so fast that at 9:00 Minute mark of video the time dilation effect of relativity is caught on film) http://www.youtube.com/watch?v=SoHeWgLvlXI
bornagain77
The fact that Oleg and Mike went beyond their natural dislike of creationists and were generous to teach me things is something I'm very appreciative of. I'm willing to endure their harsh comments about me because they have scientific knowledge that is worth learning and passing on to everyone. scordova
OlegT helped you? Is this the same olegt that now quote-mines you for brownie points? olegt's quote-mine earns him 10 points (out of 10) on the low-integrity scale Joe
SC: Please note the Macro-micro info gap issue I have highlighted above. KF kairosfocus
F/N; Please note how I speak of a sampling theory result on a config space, which is independent of precise probability calculations; we have only a reasonable expectation to pick up the bulk of the distribution. Remember we are sampling on the order of one straw to a cubical hay bale 1,000 light years on the side, i.e comparably thick to our Galaxy. KF kairosfocus
EA: When the equilibria are as unfavourable as they are, a faster reaction rate will favour breakdown, as is seen from how we refrigerate to preserve. In effect around room temp, activation processes double for every 8 K increase in temp. And, the rate of state sampling used in the FSCI calc at 500 bits as revised is actually that for the fastest ionic reactions, not the slower rates appropriate to organic ones. For 1,000 bits, we are using Planck times which are faster than anything else physical. The limits are conservative. KF kairosfocus
Shannon entropy Joe
Regarding the "Add Energy" argument. Set off an source equal in energy and power to an atomic bomb -- the results are predictable in terms of the designs (or lack thereof) that will emerge in the aftermath. That is an example where Entropy increases, but so does disorder. The problem, as illustrated with the 500-coins, is that Shannon Entropy and Thermodynamic Entropy have some independence from the notions of disorder. A designed system can have 500 bits of Shannon entropy but so can an undesigned system. Having 500 bits of Shannon entropy says little (in and of itself) whether something is desiged. An independent specification is needed to identify a design, the entropy score is only a part. We can have: 1. entropy rise and more disorder 2. entropy rise and more order 3. entropy rise and more disorganization 4 entropy rise and more organization 5. entropy rise and destroying design 6. entropy rise and creating design We can't make a general statement about what will happen to a design or a disordered system merely because the entropy rises. There are too many other variables to account for before we can say something useful. scordova
kf:
What advocates of this do not usually disclose, is that raw injection of energy tends to go to heat, i.e. to dramatic rise in the number of possible configs, given the combinational possibilities of so many lumps of energy dispersed across so many mass-particles. That is, MmIG will strongly tend to RISE on heating.
Interesting thought and worth considering. I think it is a useful point to bring up when addressing the "open system" red herring put forth by some OOL advocates, but at the end of the day it is really a rounding error on the awful probabilities that already exist. Thus, it probably makes sense to mention it in passing ("Adding energy without direction can actually make things worse.") if someone is pushing the "just add energy" line of thought, but then keep the attention focused squarely on the heart of the matter. Eric Anderson
mahuna I assume you have absolutely no experience with the specification and development of new systems.
Before becoming a financeer I was an engineer. I have 3 undergraduate degrees in electrical engineering and computer science and mathematics and a graduate engineering degree in applied physics. Of late I try to minimize mentioning it because there are so many things I don't understand which I ought to with that level of academic exposure. I fumble through statistical mechanics and thermodynamics and even basic math. I have to solicit expertise on these matters, and I have to admit that I'm wrong many times or don't know something, or misunderstand something -- and willingness to admit mistakes or lack of understanding is a quality which I find lacking among many of my creationist brethren, and even worse among evolutionary biologists. I worked on aerospace systems, digital telephony, unmanned aerial vehicles, air traffic control systems, security systems. I've written engineering specifications and carried them out. Thus
I assume you have absolutely no experience with the specification and development of new systems.
is utterly wrong and a fabrication of your own imagination. Besides, my experience is irrelevant to this discussion. At issue are the ideas and calculations. Do you have any comment on my calculations of Shannon entropy or the other entropy scores for the objects listed? scordova
Complete and utter nonsense. I assume you have absolutely no experience with the specification and development of new systems. A baseball's design is refined to eliminate every single ounce of weight or space that does not satisfy the requirements for a baseball. An airliner's design is refined to eliminate every single ounce of weight or space that does not satifsy the requirements for an airliner. But the airliner is much more complex than the baseball and didn't get that way by accident. I assume that you assume that an entropic design is launched by its designers like a Mars probe but expected to change/evolve after launch (by increasing its entropy). But as far as we know, most biologic systems are remarkably stable in their designs (um, the oldest known bat fossils are practically identical to modern bats). In "The Edge of Evolution", Behe in fact bases his argument against Evolution on the fact that there are measurably distinct levels of complexity in biologic systems, and that no known natural mechanism, most especially random degradation of the original design, will get you from a Level 2 system to a more complex Level 3 system. mahuna
gpuccio, In light of your very insightful criticism, I amended the OP as follows:
inference : designed (with qualification, see note below) .... NOTE: [UPDATE and correction: gpuccio was kind enough to point out that in the case of File_B, the design inference isn't necessarily warranted. It's possible an accident or programming error or some other reason could make all the bits 1. It would only be designed if that was the designer's intention.]
scordova
Sal: Great post!
Thank you!
A few comments: a) Shannon entropy is the basis for what we usually call the “complexity” of a digital string.
In Bill Dembski's literature, yes. Some other's will use a different metric for complexitly, like Algorithmic complexity. Phil Johnson and Stephen Meyer actually refer to algorithmic complexity if you read what they say carefully. In my previously less enlightened writings on the net I used algorithmic complexity. The point is, this confusion needs a little bit of remedy. Rather than use the word "complexity" it is easier to say what actual metric one is working from. CSI is really based on Shannon Entropy not algorithmic or thermodynamic entropy.
b) Regarding the exmaple in: File_B : 1 gigabit of all 1?s Shannon Entropy: 1 gigabit Algorithmic Entropy (Kolmogorov Complexity): low Organizational characteristics: highly organized inference : designed I would say that the inference of design is not necessarily warrnted.
Yes, thank you. I'll have to revisit this example. It's possible a programmer had the equivalent of stuck key's. I'll update the post accordingly. That's why I post stuff like this at UD, to help clean up my own thoughts. scordova
F/N: Let's do some boiling down, for summary discussion in light of the underlying matters above and in onward sources:
1: In communication situations, we are interested in information we have in hand, given certain identifiable signals (which may be digital or analogue, but can be treated as digital WLOG) 2: By contrast, in the thermodynamics situation, we are interested in the Macro-micro info gap [MmIG], i.e the "missing info" on the ultra-microscopic state of a system, given the lab-observable state of the system. 3: In the former, the inference that we have a signal, not noise, is based on an implicit determination that noise is not credibly likely to be lucky enough to mimic the signal, given the scope of the space of possible configs, vs the scope of apparently intelligent signals. 4: So, we confidently and routinely make that inference to intelligent signal not noise on receiving an apparent signal of sufficient complexity, and indeed define a key information theory metric signal to noise power ratio, on the characteristic differences between the typical observable characteristics of signals and noise. 5: Thus, we are routinely inferring that signals involving FSCO/I are not improbable on intelligent action (intelligently directed organising work, IDOW) but that they are so maximally improbable on "lucky noise" that we typically assign what looks like typical signals to real signals, and what looks like noise to noise on a routine and uncontroversial basis. 6: In the context of spontaneous OOL etc, we are receiving a signal in the living cell, which is FSCO/I rich. 7: But because there is a dominant evo mat school of thought that assumes or infers that at OOL no intelligence was existing or possible to direct organising work, it is presented as if it were essentially unquestionable knowledge, that without IDOW, FSCO/I arose. 8: In other words, despite never having observed FSCO/I arising in this way and despite the implications of the infinite monkeys/ needle in haystack type analysis, that such is essentially unobservable on the gamut of our solar system or the observed cosmos, this ideological inference is presented as if it were empirically well grounded knowledge. 9: This is unacceptable, for good reasons of avoiding question-begging. 10: By sharpest contrast, on the very same principles of inference to best current explanation of the past in light of dynamics of cause and effect in the present that we can observe as leaving characteristic signs that are comparable to traces in deposits from the past or from remote reaches of space [astrophysics], design theorists infer from the sign, FSCO/I to its cause in the remote past etc being -- per best explanation on empirical warranting grounds -- being design, or as I am specifying for this discussion: IDOW.
Let us see how this chain of reasoning is handled, here and elsewhere. KF kairosfocus
It is interesting to note that in the building of better random number generators for computer programs, a better source of entropy is required:
Cryptographically secure pseudorandom number generator Excerpt: From an information theoretic point of view, the amount of randomness, the entropy that can be generated is equal to the entropy provided by the system. But sometimes, in practical situations, more random numbers are needed than there is entropy available. http://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator
And Indeed we find:
Thermodynamics – 3.1 Entropy Excerpt: Entropy – A measure of the amount of randomness or disorder in a system. http://www.saskschools.ca/curr_content/chem30_05/1_energy/energy3_1.htm
And the maximum source of randomness in the universe is found to be,,,
Entropy of the Universe - Hugh Ross - May 2010 Excerpt: Egan and Lineweaver found that supermassive black holes are the largest contributor to the observable universe’s entropy. They showed that these supermassive black holes contribute about 30 times more entropy than what the previous research teams estimated. http://www.reasons.org/entropy-universe Roger Penrose - How Special Was The Big Bang? “But why was the big bang so precisely organized, whereas the big crunch (or the singularities in black holes) would be expected to be totally chaotic? It would appear that this question can be phrased in terms of the behaviour of the WEYL part of the space-time curvature at space-time singularities. What we appear to find is that there is a constraint WEYL = 0 (or something very like this) at initial space-time singularities-but not at final singularities-and this seems to be what confines the Creator’s choice to this very tiny region of phase space.”
,,, there is also a very strong case to be made that the cosmological constant in General Relativity, the extremely finely tuned 1 in 10^120 expansion of space-time, drives, or is deeply connected to, entropy as measured by diffusion:
Big Rip Excerpt: The Big Rip is a cosmological hypothesis first published in 2003, about the ultimate fate of the universe, in which the matter of universe, from stars and galaxies to atoms and subatomic particles, are progressively torn apart by the expansion of the universe at a certain time in the future. Theoretically, the scale factor of the universe becomes infinite at a finite time in the future. http://en.wikipedia.org/wiki/Big_Rip
Thus, though neo-Darwinian atheists may claim that evolution is as well established as Gravity, the plain fact of the matter is that General Relativity itself, which is by far our best description of Gravity, testifies very strongly against the entire concept of 'random' Darwinian evolution. also of note, quantum mechanics, which is even stronger than general relativity in terms of predictive power, has a very different 'source for randomness' which sets it as diametrically opposed to materialistic notion of randomness:
Can quantum theory be improved? – July 23, 2012 Excerpt: However, in the new paper, the physicists have experimentally demonstrated that there cannot exist any alternative theory that increases the predictive probability of quantum theory by more than 0.165, with the only assumption being that measurement (conscious observation) parameters can be chosen independently (free choice, free will assumption) of the other parameters of the theory.,,, ,, the experimental results provide the tightest constraints yet on alternatives to quantum theory. The findings imply that quantum theory is close to optimal in terms of its predictive power, even when the predictions are completely random. http://phys.org/news/2012-07-quantum-theory.html
Needless to say, finding ‘free will conscious observation’ to be ‘built into’ quantum mechanics as a starting assumption, which is indeed the driving aspect of randomness in quantum mechanics, is VERY antithetical to the entire materialistic philosophy which demands randomness as the driving force of creativity! Could these two different sources of randomness in quantum mechanics and General relativity be one of the primary reasons of their failure to be unified??? Further notes, Boltzman, as this following video alludes to,,,
,,,being a materialist, thought of randomness, entropy, as 'unconstrained', as would be expected for someone of the materialistic mindset. Yet Planck, a Christian Theist, corrected that misconception of his:
The Austrian physicist Ludwig Boltzmann first linked entropy and probability in 1877. However, the equation as shown, involving a specific constant, was first written down by Max Planck, the father of quantum mechanics in 1900. In his 1918 Nobel Prize lecture, Planck said:This constant is often referred to as Boltzmann's constant, although, to my knowledge, Boltzmann himself never introduced it – a peculiar state of affairs, which can be explained by the fact that Boltzmann, as appears from his occasional utterances, never gave thought to the possibility of carrying out an exact measurement of the constant. Nothing can better illustrate the positive and hectic pace of progress which the art of experimenters has made over the past twenty years, than the fact that since that time, not only one, but a great number of methods have been discovered for measuring the mass of a molecule with practically the same accuracy as that attained for a planet. http://www.daviddarling.info/encyclopedia/B/Boltzmann_equation.html
Related notes:
"It from bit symbolizes the idea that every item of the physical world has at bottom - at a very deep bottom, in most instances - an immaterial source and explanation; that which we call reality arises in the last analysis from the posing of yes-no questions and the registering of equipment-evoked responses; in short, that things physical are information-theoretic in origin." John Archibald Wheeler Zeilinger's principle Zeilinger's principle states that any elementary system carries just one bit of information. This principle was put forward by Austrian physicist Anton Zeilinger in 1999 and subsequently developed by him to derive several aspects of quantum mechanics. Some have reasoned that this principle, in certain ways, links thermodynamics with information theory. [1] http://www.eoht.info/page/Zeilinger%27s+principle "Is there a real connection between entropy in physics and the entropy of information? ....The equations of information theory and the second law are the same, suggesting that the idea of entropy is something fundamental..." Tom Siegfried, Dallas Morning News, 5/14/90 - Quotes attributed to Robert W. Lucky, Ex. Director of Research, AT&T, Bell Laboratories & John A. Wheeler, of Princeton & Univ. of TX, Austin in the article In the beginning was the bit - New Scientist Excerpt: Zeilinger's principle leads to the intrinsic randomness found in the quantum world. Consider the spin of an electron. Say it is measured along a vertical axis (call it the z axis) and found to be pointing up. Because one bit of information has been used to make that statement, no more information can be carried by the electron's spin. Consequently, no information is available to predict the amounts of spin in the two horizontal directions (x and y axes), so they are of necessity entirely random. If you then measure the spin in one of these directions, there is an equal chance of its pointing right or left, forward or back. This fundamental randomness is what we call Heisenberg's uncertainty principle. http://www.quantum.at/fileadmin/links/newscientist/bit.html Is it possible to find the radius of an electron? The honest answer would be, nobody knows yet. The current knowledge is that the electron seems to be a 'point particle' and has refused to show any signs of internal structure in all measurements. We have an upper limit on the radius of the electron, set by experiment, but that's about it. By our current knowledge, it is an elementary particle with no internal structure, and thus no 'size'.
bornagain77
F/N 2: We should bear in mind that information arises when we move from an a priori state to an a posteriori one where with significant assurance we are in a state that is to some degree or other surprising. Let me clip my always linked note, here on:
From a human point of view the word 'communication' conveys the idea of one person talking or writing to another in words or messages . . . through the use of words derived from an alphabet [NB: he here means, a "vocabulary" of possible signals]. Not all words are used all the time and this implies that there is a minimum number which could enable communication to be possible. In order to communicate, it is necessary to transfer information to another person, or more objectively, between men or machines. This naturally leads to the definition of the word 'information', and from a communication point of view it does not have its usual everyday meaning. Information is not what is actually in a message but what could constitute a message. The word could implies a statistical definition in that it involves some selection of the various possible messages. The important quantity is not the actual information content of the message but rather its possible information content. This is the quantitative definition of information and so it is measured in terms of the number of selections that could be made. Hartley was the first to suggest a logarithmic unit . . . and this is given in terms of a message probability. [p. 79, Signals, Edward Arnold. 1972. Bold emphasis added. Apart from the justly classical status of Connor's series, his classic work dating from before the ID controversy arose is deliberately cited, to give us an indisputably objective benchmark.]
A baseline for discussion. KF kairosfocus
F/N: I have put the above comment up with a diagram here. kairosfocus
Sal: Great post! A few comments: a) Shannon entropy is the basis for what we usually call the "complexity" of a digital string. b) Regarding the exmaple in: File_B : 1 gigabit of all 1?s Shannon Entropy: 1 gigabit Algorithmic Entropy (Kolmogorov Complexity): low Organizational characteristics: highly organized inference : designed I would say that the inference of design is not necessarily warrnted. According to the explanatory filter, in the presence of this kind of compressible order we must first ascertain that no deterministic effect is the cause of the apparent order. IOWs, many simple deterministic causes could explain a series of 1s, however long. Obviously, such a scenario would imply that the system that generates the string is not random, or that the probabilities of 0 and 1 are extremely different. I agree that, if we have assurance that the system is really random and the probabilities are as described, then a long series of 1 allows the design inference. c) A truly pseudo-random string, which has no formal evidence of order (no compressibility), like the jpeg file, but still conveys very specific information, is certainly the best scenario for design inference. Indeed, as far as I know, no deterministic system can explain the emergence of that kind of object. d) Regarding the problem of specification, I paste here what I posted yesterday in another thread, as I believe it is pertinent to the discussion here: "I suppose much confusion derives from Shannon’s theory, which is not, and never has been, a theory about information, but is often considered as such. Contemporary thought, in the full splendor of its dogmatic reductionism, has done its best to ignore the obvious connection between information and meaning. Everybody talks about information, but meaning is quite a forbidden word. As if the two things could be separated! I have discussed for days here with darwinists just trying to have them admit that sucg a thing as “function” does exist. Another forbidden word. And even IDist often are afraid to admit that meaning and function cannot even be defined if we do not refer to a conscious being. I have challenged evrybody I know to give a definition, any definition, of meaning, function and intent without recurring to conscious experience. How strange, the same concepts on which all our life, and I would say also all our science and knowledge, are based, have become forbidden in modern thought. And consciousness itself, what we are, the final medium that cognizes everything, can scarcely be mentioned, if not to affirm that it is an unscientific concept, or even better a concept completely reducible to non conscious aggregations of things (!!!). The simple truth is: there is no cognition, no science, no knowledge, without the fundamental intuition of meaning. And that intuition is a conscious event, and nothing else. There is no understanding of meaning in stones, rivers or computers. Only in conscious beings. And information is only a way to transfer menaing from one conscious being to another. Through material systems, that carry the meaning, but have no understanding of it. That’s what Shannon considered: what is necessary to transfer information through a material system. In that context, meaning is not relevant, because what we are measuring is only a law of transmission. The same is true in part for ID. The measure of complexity is a Shannon measure, it has nothing to do with meaning. A random string can be as complex as a meaningful string. But the concept of specification does relate to meaning, in one of its many aspects, for instance as function. The beautiful simplicity of ID theory is that it measures the complexity necessary to convey a specific meaning. That is simple and beautiful, beacuse it connects the quantitative concept of Shannon complexity to the qualitative aspect of meaning and function." gpuccio