Uncommon Descent Serving The Intelligent Design Community

A Designed Object’s Entropy Must Increase for Its Design Complexity to Increase – Part 1

Share
Facebook
Twitter
LinkedIn
Flipboard
Print
Email

The common belief is that adding disorder to a designed object will destroy the design (like a tornado passing through a city, to paraphrase Hoyle). Now if increasing entropy implies increasing disorder, creationists will often reason that “increasing entropy of an object will tend to destroy its design”. This essay will argue mathematically that this popular notion among creationists is wrong.

The correct conception of these matters is far more nuanced and almost the opposite of (but not quite) what many creationists and IDists believe. Here is the more correct view of entropy’s relation to design (be it man-made or otherwise):

1. increasing entropy can increase the capacity for disorder, but it doesn’t necessitate disorder

2. increasing an object’s capacity for disorder doesn’t imply that the object will immediately become more disordered

3. increasing entropy in a physical object is a necessary (but not sufficient) condition for increasing the complexity of the design

4. contrary to popular belief, a complex design is a high entropy design, not a low entropy design. The complex organization of a complex design is made possible (and simultaneously improbable) by the high entropy the object contains.

5. without entropy there is no design

If there is one key point it is: Entropy makes design possible but simultaneously improbable. And that is the nuance that many on both sides of the ID/Creation/Evolution controversy seem to miss.

The notion of entropy is foundational to physics, engineering, information theory and ID. These essays are written to provide a discussion on the topic of entropy and its relationship to other concepts such as uncertainty, probability, microstates, and disorder. Much of what is said will go against popular understanding, but the aim is to make these topics clearer. Some of the math will be in a substantially simplified form, so apologies in advance to the formalists out there.

Entropy may refer to:

1. Thermodynamic (Statistical Mechanics) entropy – measured in Joules/Kelvin, dimensionless units, degrees of freedom, or (if need be) bits

2. Shannon entropy – measured in bits or dimensionless units

3. Algorithmic entropy or Kolmogorov complexity – measured also in bits, but deals with the compactness of a representation. A file that can be compressed substantially has low algorithmic entropy, whereas files which can’t be compressed evidence high algorithmic entropy (Kolmogorov complexity). Both Shannon entropy and algorithmic entropies are within the realm of information theory, but by default, unless otherwise stated, most people associate Shannon entropy as the entropy in information theory.

4. disorder in the popular sense – no real units assigned, often not precise enough to be of scientific or engineering use. I’ll argue the term “disorder” is a misleading way to conceptualize entropy. Unfortunately, the word “disorder” is used even in university science books. I will argue mathematically why this is so…

The reason the word entropy is used in the disciplines of Thermodynamics, Statistical Mechanics and Information Theory is that there are strong mathematical analogies. The evolution of the notion of entropy began with Clausius who also coined the term for thermodynamics, then Boltzmann and Gibbs related Clausius’s notions of entropy to Newtonian (Classical) Mechanics, then Shannon took Boltzmann’s math and adapted it to information theory, and then Landauer brought things back full circle by tying thermodynamics to information theory.

How entropy became equated with disorder, I do not know, but the purpose of these essays is to walk through actual calculations of entropy and allow the reader to decide for himself whether disorder can be equated with entropy. My personal view is that Shannon entropy and Thermodynamic entropy cannot be equated with disorder, even though the lesser-known algorithmic entropy can. So in general entropy should not be equated with disorder. Further, the problem of organization (which goes beyond simple notions of order and entropy) needs a little more exploration. Organization sort of stands out as a quality that seems difficult to assign numbers to.

The calculations that follow are to give an illustration how I arrived at some my conclusions.

First I begin with calculating Shannon entropy for simple cases. Thermodynamic entropy will be covered in the Part II.

Bill Dembski actually alludes to Shannon entropy in his latest offering on Conservation of Information Made Simple

In the information-theory literature, information is usually characterized as the negative logarithm to the base two of a probability (or some logarithmic average of probabilities, often referred to as entropy).

William Dembski
Conservation of Information Made Simple

To elaborate on what Bill said, if we have a fair coin, it can exist in two microstates: heads (call it microstate 1) or tails (call it microstate 2).

After a coin flip, the probability of the coin emerging in microstate 1 (heads) is 1/2. Similarly the probability of the coin emerging in microstate 2 (tails) is 1/2. So let me tediously summarize the facts:

N = Ω(N) = Ω = Number of microstates of a 1-coin system = 2

x1 = microstate 1 = heads
x2 = microstate 2 = tails

P(x1) = P(microstate 1)= P(heads) = probability of heads = 1/2
P(x2) = P(microstate 2)= P(tails) = probability of tails = 1/2

Here is the process for calculating the Shannon Entropy of a 1-coin information system starting with Shannon’s famous formula:

where I is the Shannon entropy (or measure of information).

This method seems a rather torturous way to calculate the Shannon entropy of a single coin. A slightly simpler method exists if we take advantage of the fact that each microstate of the coin (heads or tails) is equiprobable, and thus conforms to the fundamental postulate of statistical mechanics, and thus we can calculate the number of bits by simply taking the logarithm of the number of microstates as is done in statistical mechanics.

Now compare this equation of the Shannon entropy in information theory

to Boltzmann entropy from statistical mechanics and thermodynamics

and even more so using different units whereby kb=1

The similarities are not an accident. Shannon’s ideas of information theory are a descendant of Boltzmann’s ideas from statistical mechanics and thermodynamics.

To explore Shannon entropy further, let us suppose we have a system of 3 distinct coins. The Shannon entropy relates the amount of information that will be gained by observing the collective state (microstate) of the 3 coins.

First we have to compute the number of microstates or ways the system of coins can be configured. I will lay them out specifically.

microstate 1 = H H H
microstate 2 = H H T
microstate 3 = H T H
microstate 4 = H T T
microstate 5 = T H H
microstate 6 = T H T
microstate 7 = T T H
microstate 8 = T T T

N = Ω(N) = Ω = Number of microstates of a 3-coin system = 8

So there are 8 microstates or outcomes the system can realize. The Shannon entropy can be calculated in the torturous way:

or simply taking the logarithm of the number of microstates:

It can be shown that for the Shannon entropy of a system of N distinct coins is equal to N bits. That is, a system with 1 coin has 1 bit of Shannon entropy, a system with 2 coins has 2 bits of Shannon entropy, a system of 3 coins has 3 bits of Shannon entropy, etc.

Notice, the more microstates there are, the more uncertainty exists that the system will be found in any given microstate. Equivalently, the more microstates there are, the more improbable the system will be found in a given microstate. Hence, sometimes entropy is described in terms of improbability or uncertainty or unpredictability. But we must be careful here, uncertainty is not the same thing as disorder. That is subtle but important distinction.

So what is the Shannon Entropy of a system of 500 distinct coins? Answer: 500 bits, or the Universal Probability Bound.

By way of extension, if we wanted to build an operating system like Windows-7 that requires gigabits of storage, we would require the computer memory to contain gigabits of Shannon entropy. This illustrates the principle that more complex designs require larger Shannon entropy to support the design. It cannot be otherwise. Design requires the presence of entropy, not absence of it.

Suppose we found that a system of 500 coins were all heads, what is the Shannon entropy of this 500-coin system? Answer: 500 bits. No matter what configuration the system is in, whether ordered (like all heads) or disordered, the Shannon entropy remains the same.

Now suppose a small tornado went through the room where the 500 coins resided (with all heads before the tornado), what is the Shannon entropy after the tornado? Same as before, 500-bits! What may arguably change is the algorithmic entropy (Kolmogorov complexity). The algorithmic entropy may go up, which simply means we can’t represent the configuration of the coins in a compact sort of way like saying “all heads” or in the Kleene notation as H*.

Amusingly, if in the aftermath of the tornado’s rampage, the room got cooler, the thermodynamic entropy of the coins would actually go down! Hence the order or disorder of the coins is independent not only of the Shannon entropy but also the thermodynamic entropy.

Let me summarize the before and after of the tornado going through the room with the 500 coins:

BEFORE : 500 coins all heads, Temperature 80 degrees
Shannon Entropy : 500 bits
Algorithmic Entropy (Kolmogorov complexity): low
Thermodynamic Entropy : some finite starting value

AFTER : 500 coins disordered
Shannon Entropy : 500 bits
Algorithmic Entropy (Kolmogorov complexity): high
Thermodynamic Entropy : lower if the temperature is lower, higher if the temperature is higher

Now to help disentangle concepts a little further consider three 3 computer files:

File_A : 1 gigabit of binary numbers randomly generated
File_B : 1 gigabit of all 1’s
File_C : 1 gigabit encrypted JPEG

Here are the characteristics of each file:

File_A : 1 gigabit of binary numbers randomly generated
Shannon Entropy: 1 gigabit
Algorithmic Entropy (Kolmogorov Complexity): high
Thermodynamic Entropy: N/A
Organizational characteristics: highly disorganized
inference : not designed

File_B : 1 gigabit of all 1’s
Shannon Entropy: 1 gigabit
Algorithmic Entropy (Kolmogorov Complexity): low
Thermodynamic Entropy: N/A
Organizational characteristics: highly organized
inference : designed (with qualification, see note below)

File_C : 1 gigabit encrypted JPEG
Shannon Entropy: 1 gigabit
Algorithmic Entropy (Kolmogorov complexity): high
Thermodynamic Entropy: N/A
Organizational characteristics: highly organized
inference : extremely designed

Notice, one cannot ascribe high levels of improbable design based on the Shannon entropy or algorithmic entropy without some qualification. Existence of improbable design depends on the existence of high Shannon entropy, but is somewhat independent of algorithmic entropy. Further, to my knowledge, there is not really a metric for organization that is separate from Kolmogorov complexity, but this definition needs a little more exploration and is beyond my knowledge base.

Only in rare cases will high Shannon entropy and low algorithmic entropy (Kolmogorov complexity) result in a design inference. One such example is 500 coins all heads. The general method to infer design (including man-made designs), is that the object:

1. has High Shannon Entropy (high improbability)
2. conforms to an independent (non-postdictive) specification

In contrast to the design of coins being all heads where the Shannon entropy is high but the algorithmic entropy is low, in cases like software or encrypted JPEG files, the design exists in an object that has both high Shannon entropy and high algorithmic entropy. Hence, the issues of entropy are surely nuanced, but on balance entropy is good for design, not always bad for it. In fact, if an object evidences low Shannon entropy, we will not be able to infer design reliably.

The reader might be disturbed at my final conclusion in as much as it grates against popular notions of entropy and creationist notions of entropy. But well, I’m no stranger to this controversy. I explored Shannon entropy in this thread because it is conceptually easier than its ancestor concept of thermodynamic entropy.

In the Part II (which will take a long time to write) I’ll explore thermodynamic entropy and its relationship (or lack thereof) to intelligent design. But in brief, a parallel situation often arises: the more complex a design, the higher its thermodynamic entropy. Why? The simple reason is that more complex designs involve more parts (molecules) and more molecules in general imply higher thermodynamic (as well as Shannon) entropy. So the question of Earth being an open system is a bit beside the point since entropy is essential for intelligent designs to exist in the first place.

[UPDATE: the sequel to this thread is in Part 2]

Acknowledgements (both supporters and critics):

1. Elizabeth Liddle for hosting my discussions on the 2nd Law at TheSkepticalZone

2. physicist Olegt who offered generous amounts of time in plugging the holes in my knowledge, particularly regarding the Liouville Theorem and Configurational Entropy

3. retired physicist Mike Elzinga for his pedagogical examples and historic anecdotes. HT: the relationship of more weight to more entropy

4. An un-named theoretical physicist who spent many hours teaching his students the principles of Statistical Mechanics and Thermodynamics

5. physicists Andy Jones and Rob Sheldon

6. Neil Rickert for helping me with Latex

7. Several others that have gone unnamed

NOTE:
[UPDATE and correction: gpuccio was kind enough to point out that in the case of File_B, the design inference isn’t necessarily warranted. It’s possible an accident or programming error or some other reason could make all the bits 1. It would only be designed if that was the designer’s intention.]

[UPDATE 9/7/2012]
Boltzmann

“In order to explain the fact that the calculations based on this assumption [“…that by far the largest number of possible states have the characteristic properties of the Maxwell distribution…”] correspond to actually observable processes, one must assume that an enormously complicated mechanical system represents a good picture of the world, and that all or at least most of the parts of it surrounding us are initially in a very ordered — and therefore very improbable — state. When this is the case, then whenever two of more small parts of it come into interaction with each other, the system formed by these parts is also initially in an ordered state and when left to itself it rapidly proceeds to the disordered most probable state.” (Final paragraph of #87, p. 443.)

That slight, innocent paragraph of a sincere man — but before modern understanding of q(rev)/T via knowledge of molecular behavior (Boltzmann believed that molecules perhaps could occupy only an infinitesimal volume of space), or quantum mechanics, or the Third Law — that paragraph and its similar nearby words are the foundation of all dependence on “entropy is a measure of disorder”. Because of it, uncountable thousands of scientists and non-scientists have spent endless hours in thought and argument involving ‘disorder’and entropy in the past century. Apparently never having read its astonishingly overly-simplistic basis, they believed that somewhere there was some profound base. Somewhere. There isn’t. Boltzmann was the source and no one bothered to challenge him. Why should they?

Boltzmann’s concept of entropy change was accepted for a century primarily because skilled physicists and thermodynamicists focused on the fascinating relationships and powerful theoretical and practical conclusions arising from entropy’s relation to the behavior of matter. They were not concerned with conceptual, non-mathematical answers to the question, “What is entropy, really?” that their students occasionally had the courage to ask. Their response, because it was what had been taught to them, was “Learn how to calculate changes in entropy. Then you will understand what entropy ‘really is’.”

There is no basis in physical science for interpreting entropy change as involving order and disorder.

Comments
PS: The most probable or equilibrium cluster of microstates consistent with a given macrostate, is the cluster that has the least information about it, and the most freedom of variation of mass and energy distribution at micro level. This high entropy state-cluster is strongly correlated with high levels of disorder, for reasons connected to the functionality constraints just above. And in fact -- never mind those who are objecting and pretending that this is not so -- it is widely known in physics that entropy is a metric of disorder, some would say it quantifies it and gives it structured physical expression in light of energy and randomness or information gap considerations.kairosfocus
September 8, 2012
September
09
Sep
8
08
2012
12:23 AM
12
12
23
AM
PDT
Mung: One more time [cf. 56 above, which clips elsewhere . . . ], let me clip Shannon, 1950/1:
The entropy is a statistical parameter which measures, in a certain sense, how much information is produced on the average for each letter of a text in the language. If the language is translated into binary digits (0 or 1) in the most efficient way, the entropy is the average number of binary digits required per letter of the original language. The redundancy, on the other hand, measures the amount of constraint imposed on a text in the language due to its statistical structure, e.g., in English the high fre-quency of the letter E, the strong tendency of H to follow T or of V to follow Q. It was estimated that when statistical effects extending over not more than eight letters are considered the entropy is roughly 2.3 bits per letter, the redundancy about 50 per cent.
Going back to my longstanding, always linked note, which I have clipped several times over the past few days, here on is how we measure info and avg info per symbol:
To quantify the above definition of what is perhaps best descriptively termed information-carrying capacity, but has long been simply termed information (in the "Shannon sense" - never mind his disclaimers . . .), let us consider a source that emits symbols from a vocabulary: s1,s2, s3, . . . sn, with probabilities p1, p2, p3, . . . pn. That is, in a "typical" long string of symbols, of size M [say this web page], the average number that are some sj, J, will be such that the ratio J/M --> pj, and in the limit attains equality. We term pj the a priori -- before the fact -- probability of symbol sj. Then, when a receiver detects sj, the question arises as to whether this was sent. [That is, the mixing in of noise means that received messages are prone to misidentification.] If on average, sj will be detected correctly a fraction, dj of the time, the a posteriori -- after the fact -- probability of sj is by a similar calculation, dj. So, we now define the information content of symbol sj as, in effect how much it surprises us on average when it shows up in our receiver: I = log [dj/pj], in bits [if the log is base 2, log2] . . . Eqn 1 This immediately means that the question of receiving information arises AFTER an apparent symbol sj has been detected and decoded. That is, the issue of information inherently implies an inference to having received an intentional signal in the face of the possibility that noise could be present. Second, logs are used in the definition of I, as they give an additive property: for, the amount of information in independent signals, si + sj, using the above definition, is such that: I total = Ii + Ij . . . Eqn 2 For example, assume that dj for the moment is 1, i.e. we have a noiseless channel so what is transmitted is just what is received. Then, the information in sj is: I = log [1/pj] = - log pj . . . Eqn 3 This case illustrates the additive property as well, assuming that symbols si and sj are independent. That means that the probability of receiving both messages is the product of the probability of the individual messages (pi *pj); so: Itot = log1/(pi *pj) = [-log pi] + [-log pj] = Ii + Ij . . . Eqn 4 So if there are two symbols, say 1 and 0, and each has probability 0.5, then for each, I is - log [1/2], on a base of 2, which is 1 bit. (If the symbols were not equiprobable, the less probable binary digit-state would convey more than, and the more probable, less than, one bit of information. Moving over to English text, we can easily see that E is as a rule far more probable than X, and that Q is most often followed by U. So, X conveys more information than E, and U conveys very little, though it is useful as redundancy, which gives us a chance to catch errors and fix them: if we see "wueen" it is most likely to have been "queen.") Further to this, we may average the information per symbol in the communication system thusly (giving in termns of -H to make the additive relationships clearer): - H = p1 log p1 + p2 log p2 + . . . + pn log pn or, H = - SUM [pi log pi] . . . Eqn 5 H, the average information per symbol transmitted [usually, measured as: bits/symbol], is often termed the Entropy; first, historically, because it resembles one of the expressions for entropy in statistical thermodynamics. As Connor notes: "it is often referred to as the entropy of the source." [p.81, emphasis added.] Also, while this is a somewhat controversial view in Physics, as is briefly discussed in Appendix 1below, there is in fact an informational interpretation of thermodynamics that shows that informational and thermodynamic entropy can be linked conceptually as well as in mere mathematical form . . .
What this last refers to is the Gibbs formulation of entropy for statistical mechanics, and its implications when the relationship between probability and information is brought to bear in light of the Macro-micro views of a body of matter. That is, when we have a body, we can characterise its state per lab-level thermodynamically significant variables, that are reflective of many possible ultramicroscopic states of constituent particles. Thus, clipping again from my always linked discussion that uses Robertson's Statistical Thermophysics, CH 1 [and do recall my strong recommendation that we all acquire and read L K Nash's elements of Statistical Thermodynamics as introductory reading):
Summarising Harry Robertson's Statistical Thermophysics (Prentice-Hall International, 1993) . . . . For, as he astutely observes on pp. vii - viii:
. . . the standard assertion that molecular chaos exists is nothing more than a poorly disguised admission of ignorance, or lack of detailed information about the dynamic state of a system . . . . If I am able to perceive order, I may be able to use it to extract work from the system, but if I am unaware of internal correlations, I cannot use them for macroscopic dynamical purposes. On this basis, I shall distinguish heat from work, and thermal energy from other forms . . .
And, in more details, (pp. 3 - 6, 7, 36, cf Appendix 1 below for a more detailed development of thermodynamics issues and their tie-in with the inference to design; also see recent ArXiv papers by Duncan and Samura here and here):
. . . It has long been recognized that the assignment of probabilities to a set represents information, and that some probability sets represent more information than others . . . if one of the probabilities say p2 is unity and therefore the others are zero, then we know that the outcome of the experiment . . . will give [event] y2. Thus we have complete information . . . if we have no basis . . . for believing that event yi is more or less likely than any other [we] have the least possible information about the outcome of the experiment . . . . A remarkably simple and clear analysis by Shannon [1948] has provided us with a quantitative measure of the uncertainty, or missing pertinent information, inherent in a set of probabilities [NB: i.e. a probability different from 1 or 0 should be seen as, in part, an index of ignorance] . . . . [deriving informational entropy, cf. discussions here, here, here, here and here; also Sarfati's discussion of debates and the issue of open systems here . . . ] H({pi}) = - C [SUM over i] pi*ln pi, [. . . "my" Eqn 6] [--> This is essentially the same as Gibbs Entropy, once C is properly interpreted and the pi's relate to the probabilities of microstates consistent with the given lab-observable macrostate of a system at a given Temp, with a volume V, under pressure P, degree of magnetisation, etc etc . . . ] [where [SUM over i] pi = 1, and we can define also parameters alpha and beta such that: (1) pi = e^-[alpha + beta*yi]; (2) exp [alpha] = [SUM over i](exp - beta*yi) = Z [Z being in effect the partition function across microstates, the "Holy Grail" of statistical thermodynamics]. . . . [H], called the information entropy, . . . correspond[s] to the thermodynamic entropy [i.e. s, where also it was shown by Boltzmann that s = k ln w], with C = k, the Boltzmann constant, and yi an energy level, usually ei, while [BETA] becomes 1/kT, with T the thermodynamic temperature . . . A thermodynamic system is characterized by a microscopic structure that is not observed in detail . . . We attempt to develop a theoretical description of the macroscopic properties in terms of its underlying microscopic properties, which are not precisely known. We attempt to assign probabilities to the various microscopic states . . . based on a few . . . macroscopic observations that can be related to averages of microscopic parameters. Evidently the problem that we attempt to solve in statistical thermophysics is exactly the one just treated in terms of information theory. It should not be surprising, then, that the uncertainty of information theory becomes a thermodynamic variable when used in proper context . . . . Jayne's [summary rebuttal to a typical objection] is ". . . The entropy of a thermodynamic system is a measure of the degree of ignorance of a person whose sole knowledge about its microstate consists of the values of the macroscopic quantities . . . which define its thermodynamic state. This is a perfectly 'objective' quantity . . . it is a function of [those variables] and does not depend on anybody's personality. There is no reason why it cannot be measured in the laboratory." . . . . [pp. 3 - 6, 7, 36; replacing Robertson's use of S for Informational Entropy with the more standard H.]
As is discussed briefly in Appendix 1, Thaxton, Bradley and Olsen [TBO], following Brillouin et al, in the 1984 foundational work for the modern Design Theory, The Mystery of Life's Origins [TMLO], exploit this information-entropy link, through the idea of moving from a random to a known microscopic configuration in the creation of the bio-functional polymers of life, and then -- again following Brillouin -- identify a quantitative information metric for the information of polymer molecules. For, in moving from a random to a functional molecule, we have in effect an objective, observable increment in information about the molecule. This leads to energy constraints, thence to a calculable concentration of such molecules in suggested, generously "plausible" primordial "soups." In effect, so unfavourable is the resulting thermodynamic balance, that the concentrations of the individual functional molecules in such a prebiotic soup are arguably so small as to be negligibly different from zero on a planet-wide scale. By many orders of magnitude, we don't get to even one molecule each of the required polymers per planet, much less bringing them together in the required proximity for them to work together as the molecular machinery of life. The linked chapter gives the details. More modern analyses [e.g. Trevors and Abel, here and here], however, tend to speak directly in terms of information and probabilities rather than the more arcane world of classical and statistical thermodynamics . . .
Now, of course, as Wiki summarises, the classic formulation of the Gibbs entropy is:
The macroscopic state of the system is defined by a distribution on the microstates that are accessible to a system in the course of its thermal fluctuations. So the entropy is defined over two different levels of description of the given system. The entropy is given by the Gibbs entropy formula, named after J. Willard Gibbs. For a classical system (i.e., a collection of classical particles) with a discrete set of microstates, if E_i is the energy of microstate i, and p_i is its probability that it occurs during the system's fluctuations, then the entropy of the system is: S = -k_B * [sum_i] p_i * ln p_i This definition remains valid even when the system is far away from equilibrium. Other definitions assume that the system is in thermal equilibrium, either as an isolated system, or as a system in exchange with its surroundings. The set of microstates on which the sum is to be done is called a statistical ensemble. Each statistical ensemble (micro-canonical, canonical, grand-canonical, etc.) describes a different configuration of the system's exchanges with the outside, from an isolated system to a system that can exchange one more quantity with a reservoir, like energy, volume or molecules. In every ensemble, the equilibrium configuration of the system is dictated by the maximization of the entropy of the union of the system and its reservoir, according to the second law of thermodynamics (see the statistical mechanics article). Neglecting correlations between the different possible states (or, more generally, neglecting statistical dependencies between states) will lead to an overestimate of the entropy[1]. These correlations occur in systems of interacting particles, that is, in all systems more complex than an ideal gas. This S is almost universally called simply the entropy. It can also be called the statistical entropy or the thermodynamic entropy without changing the meaning. Note the above expression of the statistical entropy is a discretized version of Shannon entropy. The von Neumann entropy formula is an extension of the Gibbs entropy formula to the quantum mechanical case. It has been shown that the Gibb's Entropy is numerically equal to the experimental entropy[2] dS = delta_Q/{T} . . .
Looks to me that this is one time Wiki has it just about dead right. Let's deduce a relationship that shows physical meaning in info terms, where (- log p_i) is an info metric, I-i, here for microstate i, and noting that a sum over i of p_i * log p_i is in effect a frequency/probability weighted average or the expected value of the log p_i expression, and also moving away from natural logs (ln) to generic logs:
S_Gibbs = -k_B * [sum_i] p_i * log p_i But, I_i = - log p_i So, S_Gibbs = k_B * [sum_i] p_i * I-i i.e. S-Gibbs is a constant times the average information required to specify the particular microstate of the system, given its macrostate, the MmIG (macro-micro info gap.
Or, as Wiki also says elsewhere:
At an everyday practical level the links between information entropy and thermodynamic entropy are not close. Physicists and chemists are apt to be more interested in changes in entropy as a system spontaneously evolves away from its initial conditions, in accordance with the second law of thermodynamics, rather than an unchanging probability distribution. And, as the numerical smallness of Boltzmann's constant kB indicates, the changes in S / kB for even minute amounts of substances in chemical and physical processes represent amounts of entropy which are so large as to be right off the scale compared to anything seen in data compression or signal processing. But, at a multidisciplinary level, connections can be made between thermodynamic and informational entropy, although it took many years in the development of the theories of statistical mechanics and information theory to make the relationship fully apparent. In fact, in the view of Jaynes (1957), thermodynamics should be seen as an application of Shannon's information theory: the thermodynamic entropy is interpreted as being an estimate of the amount of further Shannon information needed to define the detailed microscopic state of the system, that remains uncommunicated by a description solely in terms of the macroscopic variables of classical thermodynamics. For example, adding heat to a system increases its thermodynamic entropy because it increases the number of possible microscopic states that it could be in, thus making any complete state description longer. (See article: maximum entropy thermodynamics.[Also,another article remarks: >>in the words of G. N. Lewis writing about chemical entropy in 1930, "Gain in entropy always means loss of information, and nothing more" . . . in the discrete case using base two logarithms, the reduced Gibbs entropy is equal to the minimum number of yes/no questions that need to be answered in order to fully specify the microstate, given that we know the macrostate.>>]) Maxwell's demon can (hypothetically) reduce the thermodynamic entropy of a system by using information about the states of individual molecules; but, as Landauer (from 1961) and co-workers have shown, to function the demon himself must increase thermodynamic entropy in the process, by at least the amount of Shannon information he proposes to first acquire and store; and so the total entropy does not decrease (which resolves the paradox).
So, immediately, the use of "entropy" in the Shannon context, to denote not H but N*H, where N is the number of symbols (thus, step by step states emitting those N symbols involved), is an error of loose reference. Similarly, by exploiting parallels in formulation and insights into the macro-micro distinction in thermodynamics, we can develop a reasonable and empirically supportable physical account of how Shannon information is a component of the Gibbs entropy narrative. Where also Gibbs subsumes the Boltzmann formulation and onward links to the lab-measurable quantity. (Nash has a useful, relatively lucid -- none of this topic is straightforward -- discussion on that.) Going beyond, once the bridge is there between information and entropy, it is there. It is not going away, regardless of how inconvenient it may be to some schools of thought. We can easily see that, for example, information is expressed in the configuration of a string, Z, of elements z1 -z2 . . . zN in accordance with a given protocol of assignment rules and interpretation & action rules etc. Where also, such is WLOG as AutoCAD etc show us that using the nodes and arcs representation and a list of structured strings that record this, essentially any object can be described in terms of a suitably configured string or collection of strings. So now, we can see that string Z (with each zi possibly taking b discrete states) may represent an island of function that expresses functionally specific complex organisation and associated information. Because of specificity to achieve and keep function, leading to a demand for matching, co-ordinated values of zi along the string, that string has relatively few of the N^b possibilities for N elements with b possible states being permissible. We are at isolated islands of specific function i.e cases E from a zone of function T in a space of possibilities W. (BTW, once b^N exceeds 500 bits on the gamut of our solar system, or 1,000 bits on the gamut of our observable cosmos, that brings to bear all the needle in the haystack, monkeys at keyboards analysis that has been repeatedly brought forth to show why FSCO/I is a useful sign of IDOW -- intelligently directed organising work -- as empirically credible cause.) We see then that we have a complex string to deal with, with sharp restrictions on possible configs, that are evident from observable function, relative to the general possibility of W = b^N possibilities. Z is in a highly informational, tightly constrained state that comes form a special zone specifiable on macro-level observable function (without actually observing Z directly). That constraint on degrees of freedom contingent on functional, complex organisation, is tantamount to saying that a highly informational state is a low entropy one, in the Gibbs sense. Going back to the expression, comparatively speaking there is not a lot of MISSING micro-level info to be specified, i.e. simply by knowing the fact of complex specified information-rich function, we know that we are in a highly restricted special Zone T in W. This immediately applies to R/DNA and proteins, which of course use string structures. It also applies tot he complex 3-D arrangement of components in the cell, which are organised in ways that foster function. And of course it applies to the 747 in a flyable condition. Such easily explains why a tornado passing through a junkyard in Seattle will not credibly assemble a 747 from parts it hits, and it explains why the raw energy and forces of the tornado that hits another formerly flyable 747, and tearing it apart, would render its resulting condition much less specified per function, and in fact result in predictable loss of function. We will also see that this analysis assumes the functional possibilities of a mass of Al, but is focussed on the issue of functional config and gives it specific thermodynamics and information theory context. (Where also, algebraic modelling is a valid mathematical analysis.) I trust this proves helpful KFkairosfocus
September 8, 2012
September
09
Sep
8
08
2012
12:18 AM
12
12
18
AM
PDT
They were not concerned with conceptual, non-mathematical answers to the question, “What is entropy, really?” that their students occasionally had the courage to ask.
Does Lambert answer that question? What is Entropy, really? So, entropy is the answer to the age-old question, why me?Mung
September 7, 2012
September
09
Sep
7
07
2012
05:52 PM
5
05
52
PM
PDT
...and when left to itself it rapidly proceeds to the most probable state.
There, I fixed it fer ya! As a bonus you get the "directionality" of entropy. Ordered and disordered gots nothing to do with it.Mung
September 7, 2012
September
09
Sep
7
07
2012
05:41 PM
5
05
41
PM
PDT
Boltzmann
“In order to explain the fact that the calculations based on this assumption [“…that by far the largest number of possible states have the characteristic properties of the Maxwell distribution…”] correspond to actually observable processes, one must assume that an enormously complicated mechanical system represents a good picture of the world, and that all or at least most of the parts of it surrounding us are initially in a very ordered — and therefore very improbable — state. When this is the case, then whenever two of more small parts of it come into interaction with each other, the system formed by these parts is also initially in an ordered state and when left to itself it rapidly proceeds to the disordered most probable state.” (Final paragraph of #87, p. 443.)
That slight, innocent paragraph of a sincere man — but before modern understanding of q(rev)/T via knowledge of molecular behavior (Boltzmann believed that molecules perhaps could occupy only an infinitesimal volume of space), or quantum mechanics, or the Third Law — that paragraph and its similar nearby words are the foundation of all dependence on “entropy is a measure of disorder”. Because of it, uncountable thousands of scientists and non-scientists have spent endless hours in thought and argument involving ‘disorder’and entropy in the past century. Apparently never having read its astonishingly overly-simplistic basis, they believed that somewhere there was some profound base. Somewhere. There isn’t. Boltzmann was the source and no one bothered to challenge him. Why should they? Boltzmann’s concept of entropy change was accepted for a century primarily because skilled physicists and thermodynamicists focused on the fascinating relationships and powerful theoretical and practical conclusions arising from entropy’s relation to the behavior of matter. They were not concerned with conceptual, non-mathematical answers to the question, “What is entropy, really?” that their students occasionally had the courage to ask. Their response, because it was what had been taught to them, was “Learn how to calculate changes in entropy. Then you will understand what entropy ‘really is’.” There is no basis in physical science for interpreting entropy change as involving order and disorder.
scordova
September 7, 2012
September
09
Sep
7
07
2012
07:58 AM
7
07
58
AM
PDT
From the OP:
How entropy became equated with disorder, I do not know ...
Arieh Ben-Naim writes: "It should be noted that Boltzmann himself was perhaps the first to use the "disorder" metaphor in his writing:
...are initially in a very ordered - therefore very improbable - state ... when left to itself it rapidly proceeds to the disordered most probable state. - Boltzmann (1964)
You should note that Boltzmann uses the terms "order" and "disorder" as qualitative descriptions of what goes on in the system. When he defines entropy, however, he uses either the number of states or probability. Indeed, there are many examples where the term "disorder" can be applied to describe entropy. For instance, mixing two gases is well described as a process leading to a higher degree of disorder. However, there are many examples for which the disorder metaphor fails."Mung
September 6, 2012
September
09
Sep
6
06
2012
10:11 AM
10
10
11
AM
PDT
As I have said above, the adoption of the term "entropy" for SMI was an unfortunate event, not because entropy is not SMI, but because SMI is not entropy!.
*SMI - Shannon's Measure of Information http://www.worldscientific.com/worldscibooks/10.1142/7694Mung
September 6, 2012
September
09
Sep
6
06
2012
09:18 AM
9
09
18
AM
PDT
OOPS, 600 + Exa BYTESkairosfocus
September 6, 2012
September
09
Sep
6
06
2012
07:47 AM
7
07
47
AM
PDT
PS: As I head out, I think an estimate of what it would take to describe the state of 1 cc of monoatomic ideal gas at 760 mm HG and 0 degrees C, i.e. 2.687 * 10^19 particles with 6 degrees of positional and momentum freedom would help us. Let us devote 32 bits -- 16 bits to get 4 hex sig figs, and a sign bit plus 15 bits for the binary exponent to each of the (x, y, z) and (P_x, P-y and P-z) co-ordinates in the phase space. We are talking about:
2.687 * 10^19 particles x 32 bits per degree of freedom x 6 degrees of freedom each _____________ 5.159 * 10^21 bits of info
That is, to describe the state of the system at a given instant, we would need 5.159 * 10^21 bits, or 644.9 * 10^18 bits. That is how many yes/no quest5ions, in teh correct order, would have to be amnswered and processed every clock tick we update. And with 10^-14 s as a reasonable chemical reaction rate, we are seeing a huge amount of required processing to keep track. As to how that would be done, that is anybody's guess.kairosfocus
September 6, 2012
September
09
Sep
6
06
2012
07:46 AM
7
07
46
AM
PDT
F/N: This from OP needs comment:
what is the Shannon Entropy of a system of 500 distinct coins? Answer: 500 bits, or the Universal Probability Bound. By way of extension, if we wanted to build an operating system like Windows-7 that requires gigabits of storage, we would require the computer memory to contain gigabits of Shannon entropy. This illustrates the principle that more complex designs require larger Shannon entropy to support the design. It cannot be otherwise. Design requires the presence of entropy, not absence of it.
Actually, in basic info theory, H strictly is a measure of average info content per element in a system or symbol in a message. Hence its being estimated on a weighted average of information per relevant element. This, I illustrated earlier from a Shannon 1950/1 paper, in comment 15 in the part 2 thread:
The entropy is a statistical parameter which measures, in a certain sense, how much information is produced on the average for each letter of a text in the language. If the language is translated into binary digits (0 or 1) in the most efficient way, the entropy is the average number of binary digits required per letter of the original language. The redundancy, on the other hand, measures the amount of constraint imposed on a text in the language due to its statistical structure, e.g., in English the high fre-quency of the letter E, the strong tendency of H to follow T or of V to follow Q. It was estimated that when statistical effects extending over not more than eight letters are considered the entropy is roughly 2.3 bits per letter, the redundancy about 50 per cent.
So, we see the context of usage here. But what happens when you have a message of N elements? In the case of a system of complexity N elements, then the cumulative, Shannon metric based information -- notice how I am shifting terms to avoid ambiguity -- is, logically, H + H + . . . H N times over, or N * H. And, as was repeatedly highlighted, in the case of the entropy of systems that are in clusters of microstates consistent with a macrostate, the thermodynamic entropy is usefully measured by and understood on terms of the Macro-micro information gap (MmIG], not on a per state or per particle basis but a cumulative basis: we know macro quantities, not the specific position and momentum of each particle, from moment to moment, which given chaos theory we could not keep track of anyway. A useful estimate per the Gibbs weighed probability sum entropy metric -- which is where Shannon reputedly got the term he used from in the first place, on a suggestion from von Neumann -- is:
>>in the words of G. N. Lewis writing about chemical entropy in 1930, "Gain in entropy always means loss of information, and nothing more" . . . in the discrete case using base two logarithms, the reduced Gibbs entropy is equal to the minimum number of yes/no questions that need to be answered in order to fully specify the microstate, given that we know the macrostate. >>
Where, Wiki gives a useful summary:
The macroscopic state of the system is defined by a distribution on the microstates that are accessible to a system in the course of its thermal fluctuations. So the entropy is defined over two different levels of description of the given system. The entropy is given by the Gibbs entropy formula, named after J. Willard Gibbs. For a classical system (i.e., a collection of classical particles) with a discrete set of microstates, if E_i is the energy of microstate i [--> Notice, summation is going to be over MICROSTATES . . . ], and p_i is its probability that it occurs during the system's fluctuations, then the entropy of the system is S_sys = - k_B [SUM over i's] P_i log p_i
Also, {- log p_i} is an information metric, I_i, i.e the information we would learn on actually coming to know that the system is in microstate i. Thus, we are taking a scaled info metric on the probabilistically weighted summmation of info in each microstate. Let us adjust: S_sys = k_B [SUM over i's] p_i * I_i This is the weighted average info per possible microstate, scaled by k_B. (Which of course is where the Joules per Kelvin come from.) In effect the system is giving us a message, its macrostate, but that message is ambiguous over the specific microstate in it. After a bit of mathematical huffing and puffing, we are seeing that the entropy is linked to the average info per possible microstate. Where this is going is of course that when a system is in a state with many possible microstates, it has enormous freedom of being in possible configs, but if the macro signals lock us down to specific states in small clusters, we need to account for how it could be in such clusters, when under reasonable conditions and circumstances, it could be easily in states that are far less specific. In turn that raises issues over IDOW. Which then points onward to FSCO/I being a sign of intelligent design. KFkairosfocus
September 6, 2012
September
09
Sep
6
06
2012
07:18 AM
7
07
18
AM
PDT
But what if you only have the encoded string to work upon, and the JPEG codec generates an apparently random string as output? How do you tell whether the output signal is truly random or that it contains a human-readable message encoded using some other protocol?
You can't tell if a string is truly the product of mindless purposeless forces (random is your word), so you have to be agnostic about that. So one must accept that one can make a false inference to randomness (such as when someone wants to be extremely stealthy and encrypt the data). If it parses with another codec that is avaiable to you, you have good reason to accept the file is designed. Beyond that, one might have other techniques such as those that team Norton Symantec used to determine that Stuxnet was the product of an incredible level of intelligent design: How Digital Detectives Deciphered Stuxnet
Several layers of masking obscured the zero-day exploit inside, requiring work to reach it, and the malware was huge — 500k bytes, as opposed to the usual 10k to 15k. Generally malware this large contained a space-hogging image file, such as a fake online banking page that popped up on infected computers to trick users into revealing their banking login credentials. But there was no image in Stuxnet, and no extraneous fat either. The code appeared to be a dense and efficient orchestra of data and commands. .... Instead, Stuxnet stored its decrypted malicious DLL file only in memory as a kind of virtual file with a specially crafted name. It then reprogrammed the Windows API — the interface between the operating system and the programs that run on top of it — so that every time a program tried to load a function from a library with that specially crafted name, it would pull it from memory instead of the hard drive. Stuxnet was essentially creating an entirely new breed of ghost file that would not be stored on the hard drive at all, and hence would be almost impossible to find. O Murchu had never seen this technique in all his years of analyzing malware. “Even the complex threats that we see, the advanced threats we see, don’t do this,” he mused during a recent interview at Symantec’s office. Clues were piling up that Stuxnet was highly professional, and O Murchu had only examined the first 5k of the 500k code. It was clear it was going to take a team to tackle it. The question was, should they tackle it? .... But Symantec felt an obligation to solve the Stuxnet riddle for its customers. More than this, the code just seemed way too complex and sophisticated for mere espionage. It was a huge adrenaline-rush of a puzzle, and O Murchu wanted to crack it. “Everything in it just made your hair stand up and go, this is something we need to look into,” he said. .... As Chien and O Murchu mapped the geographical location of the infections, a strange pattern emerged. Out of the initial 38,000 infections, about 22,000 were in Iran. Indonesia was a distant second, with about 6,700 infections, followed by India with about 3,700 infections. The United States had fewer than 400. Only a small number of machines had Siemens Step 7 software installed – just 217 machines reporting in from Iran and 16 in the United States. The infection numbers were way out of sync with previous patterns of worldwide infections — such as what occurred with the prolific Conficker worm — in which Iran never placed high, if at all, in infection stats. South Korea and the United States were always at the top of charts in massive outbreaks, which wasn’t a surprise since they had the highest numbers of internet users. But even in outbreaks centered in the Middle East or Central Asia, Iran never figured high in the numbers. It was clear the Islamic Republic was at the center of the Stuxnet infection. The sophistication of the code, plus the fraudulent certificates, and now Iran at the center of the fallout made it look like Stuxnet could be the work of a government cyberarmy — maybe even a United States cyberarmy.
And that illustrates how a non-random string in a computer might be deduced as the product of some serious ID.scordova
September 6, 2012
September
09
Sep
6
06
2012
07:00 AM
7
07
00
AM
PDT
But even if you didn’t know in advance, the fact that a JPEG decoder could produce a meaningful image proves only that the message was encoded using the JPEG protocol.
And JPEG encoders are intellignetly deisgned, so the files generated are still products of intelligent design.
A magnificent feat of inference.
Indeed.
It might be interesting if you could prove that the message originated from a non-human source
Humans can make JPEGs, so no need to invoke non-human sources.scordova
September 6, 2012
September
09
Sep
6
06
2012
06:32 AM
6
06
32
AM
PDT
TA: Why not look over in the next thread 23 - 24 (with 16 in context as background)? Kindly explain the behaviour of the black box that emits ordered vs random vs meaningful text strings of 502 bits: || BLACK BOX || –> 502 bit string As in, explain to us, how emitting the string of ASCII characters for the first 72 or so letters of this post is not an excellent reason to infer to design as the material cause of the organised string. As in, intelligently directed organising work, which I will label for convenience, IDOW. Can you justify a claim that lucky noise plus mechanical necessity adequately explains such an intelligible string, in the teeth of what sampling theory tells us on the likely outcome of samples on the gamut of the 10^57 atoms of the solar system for 10^17 s, at about 10^14 sa/s -- comparable to fast chemical ionic reaction rates -- relative to the space of possible configs of 500 bits. (As in 1 straw-size to a cubical hay bale of 1,000 LY on the side about as thick as our galaxy.) As in, we have reason to infer on FSCO/I as an empirically reliable sign of design, no great surprise, never mind your recirculation of long since cogently answered objections. (NB: This is what often happens when a single topic gets split up by using rapid succession of threads with comments. That is why I posted a reference thread, with a link back and no comments.) KFkairosfocus
September 6, 2012
September
09
Sep
6
06
2012
05:45 AM
5
05
45
AM
PDT
No timothya- I don't think so. It is obvious. And not one of those biologists can produce any evidence that demonstrates otherwise.Joe
September 6, 2012
September
09
Sep
6
06
2012
05:33 AM
5
05
33
AM
PDT
Joe I am clear that you think so. You are in disagreement with almost every practising biologist in the world of science. But that is your choice. In the meantime, can we focus on Sal's proposal?timothya
September 6, 2012
September
09
Sep
6
06
2012
05:24 AM
5
05
24
AM
PDT
timothya- there isn't any evidence that natural selection is non-random- just so that we are clear.Joe
September 6, 2012
September
09
Sep
6
06
2012
05:20 AM
5
05
20
AM
PDT
Joe posted this:
And timothya- I am still waiting for evidence that natural selection is non-random….
As far as it matters, you have already had your answer in a different thread. This thread seems to be focussed on the "how to identify designedness", so perhaps we should stick to that subject.timothya
September 6, 2012
September
09
Sep
6
06
2012
05:15 AM
5
05
15
AM
PDT
Joe posted this:
I’m saying that if you find a file on a compuetr then it a given some agency put it there.
Brilliant insight. Users of computers generate artefacts that are stored in a form determined by the operating system of the computer that they are using (in turn determined by the human designers of the operating system involved). I would be a little surprised if it proved to be otherwise. However, the reliable transformation of input data to stored data in computer storage doesn't help Sal with his problem of how to assign "designedness" to an arbitrary string of input data. He has to show that there is a reliable way to distinguish between a genuinely random string and a pseudorandom string that is hiding a human-readable message, when all he has to go on is the string itself, with no prior knowledge. If he has such a method, I would be fascinated to know what it is.timothya
September 6, 2012
September
09
Sep
6
06
2012
05:06 AM
5
05
06
AM
PDT
And timothya- I am still waiting for evidence that natural selection is non-random....Joe
September 6, 2012
September
09
Sep
6
06
2012
04:51 AM
4
04
51
AM
PDT
I'm saying that if you find a file on a compuetr then it a given some agency put it there.Joe
September 6, 2012
September
09
Sep
6
06
2012
04:50 AM
4
04
50
AM
PDT
Joe posted this:
I would bet that both strings are the product of agency involvement as blind and undirected processes cannot construct a file.
Forget the container and consider the thing contained (I mean, really, do I have to define every parameter of the discussion?). Scientists sensing signals from a pulsar store the results in a computer "file" via a series of truth-preserving transformations (light data to electronics to magnetic marks on a hard drive). Are you arguing that the stored data does not correlate reliably to the original sense data?timothya
September 6, 2012
September
09
Sep
6
06
2012
04:44 AM
4
04
44
AM
PDT
Waiting for Sal's response, I noticed that he posted this:
The fact that I knew File C was a JPEG suggests that I had some advanced knowledge of the file being designed. And even if I didn’t know that in advance, the fact that it could be parsed and processed as a JPEG indicates that it is organized.
Exactly. You knew in advance that the file was JPEG-encoded. But even if you didn't know in advance, the fact that a JPEG decoder could produce a meaningful image proves only that the message was encoded using the JPEG protocol. A magnificent feat of inference. It might be interesting if you could prove that the message originated from a non-human source. Otherwise not. But what if you only have the encoded string to work upon, and the JPEG codec generates an apparently random string as output? How do you tell whether the output signal is truly random or that it contains a human-readable message encoded using some other protocol? If I understand your original post, you claim that design is detectable from the pattern of the encoded message, independent of its mode of encoding.timothya
September 6, 2012
September
09
Sep
6
06
2012
04:34 AM
4
04
34
AM
PDT
If you have a means of distinguishing between File X, (which contains a genuine random strong), and File Y (which contains a pseudorandom random string encoding a human-readable sentence), then fill your boots and publish the method.
I would bet that both strings are the product of agency involvement as blind and undirected processes cannot construct a file.Joe
September 6, 2012
September
09
Sep
6
06
2012
04:18 AM
4
04
18
AM
PDT
If you have a means of distinguishing between File X, (which contains a genuine random strong), and File Y (which contains a pseudorandom random string encoding a human-readable sentence), then fill your boots and publish the method. The sound you can hear is that of computer security specialists the world over shifting uncomfortably in their seats. Or perhaps of computer security specialists laughing their faces off. The point is this: if you want to infer "design" solely from the evidence (of the contents of the files, with no a priori knowledge of their provenance), then what is your method?timothya
September 6, 2012
September
09
Sep
6
06
2012
03:12 AM
3
03
12
AM
PDT
If you didn’t know in advance what the origin of File A and File C were, then you would have no useful evidence from the contents of the two files to decide that one was “highly disorganised” and the other was “highly organised”.
The fact that I knew File C was a JPEG suggests that I had some advanced knowledge of the file being designed. And even if I didn't know that in advance, the fact that it could be parsed and processed as a JPEG indicates that it is organized. The fact that I specified in advance that FILE A was created by a random number generator ensures a high probability it will not be designed. File B had to be restated with qualification as gpuccio pointed out. The inference of design or lack thereof was based on advanced prior knowledge, not some explantory filter after the fact.scordova
September 6, 2012
September
09
Sep
6
06
2012
02:30 AM
2
02
30
AM
PDT
Correction, I posted this:
Your argument appears to say that if a system transmits a constant signal, then it must be organised.
I meant to use the term from your post that the valid inference for File B was that the file contents were designed. Clearly a gigabit of "ones" is organised in the sense that it has an evident pattern.timothya
September 6, 2012
September
09
Sep
6
06
2012
02:30 AM
2
02
30
AM
PDT
Sal posted this:
Now to help disentangle concepts a little further consider three 3 computer files:
File_A : 1 gigabit of binary numbers randomly generated File_B : 1 gigabit of all 1?s File_C : 1 gigabit encrypted JPEG Here are the characteristics of each file: File_A : 1 gigabit of binary numbers randomly generated Shannon Entropy: 1 gigabit Algorithmic Entropy (Kolmogorov Complexity): high Thermodynamic Entropy: N/A Organizational characteristics: highly disorganized inference : not designed File_B : 1 gigabit of all 1?s Shannon Entropy: 1 gigabit Algorithmic Entropy (Kolmogorov Complexity): low Thermodynamic Entropy: N/A Organizational characteristics: highly organized inference : designed (with qualification, see note below) File_C : 1 gigabit encrypted JPEG Shannon Entropy: 1 gigabit Algorithmic Entropy (Kolmogorov complexity): high Thermodynamic Entropy: N/A Organizational characteristics: highly organized inference : extremely designed
Please tell me that you are joking. If you didn't know in advance what the origin of File A and File C were, then you would have no useful evidence from the contents of the two files to decide that one was "highly disorganised" and the other was "highly organised". Hint: the purpose of encryption is to make the contents of the file approach as closely as possible to a randomly generated string. File B supports an inference of "highly organised"? How? Why? What if the ground state of the signal is just the continuous emission of something interpreted digitally as "ones" (or zeroes" for that matter). Your argument appears to say that if a system transmits a constant signal, then it must be organised.timothya
September 6, 2012
September
09
Sep
6
06
2012
02:20 AM
2
02
20
AM
PDT
FROM MUNG: No Sal, 500 pennies gets you 500 bits of copper plated zinc, not 500 bits of information (or Shannon entropy).
Contrast to Bill Dembski's recent article:
FROM BILL DEBMSKI In the information-theory literature, information is usually characterized as the negative logarithm to the base two of a probability (or some logarithmic average of probabilities, often referred to as entropy). This has the effect of transforming probabilities into bits and of allowing them to be added (like money) rather than multiplied (like probabilities). Thus, a probability of one-eighths, which corresponds to tossing three heads in a row with a fair coin, corresponds to three bits,
I just did a comparable calculation more elaborately, and you missed it. Instead of tossing a single coin 3 times, I had 3 coins tossed 1 time.
FROM WIKI A single toss of a fair coin has an entropy of one bit. A series of two fair coin tosses has an entropy of two bits. The entropy rate for the coin is one bit per toss
I wrote the analogous situation, except insted of making multiple tosses of a single coin, I did the formula for single tosses of multiple coins. The Shannon entropy is analogous.
I wrote: It can be shown that for the Shannon entropy of a system of N distinct coins is equal to N bits. That is, a system with 1 coin has 1 bit of Shannon entropy, a system with 2 coins has 2 bits of Shannon entropy, a system of 3 coins has 3 bits of Shannon entropy, etc.
scordova
September 5, 2012
September
09
Sep
5
05
2012
10:18 PM
10
10
18
PM
PDT
Entropy:
The notion of entropy is foundational to physics, engineering, information theory and ID. These essays are written to provide a discussion on the topic of entropy and its relationship to other concepts such as uncertainty, probability, microstates, and disorder. Much of what is said will go against popular understanding, but the aim is to make these topics clearer.
ok, so what is entropy?
First I begin with calculating Shannon entropy for simple cases.
ok, but first, what is "Shannon entropy"?
2. Shannon entropy – measured in bits or dimensionless units
Telling me it's measured in bits doesn't tell me what "it" is.
I is the Shannon entropy (or measure of information).
So "Shannon entropy" is a measure of information?
Hence, sometimes entropy is described in terms of improbability or uncertainty or unpredictability.
So Shannon entropy is a measure of what we don't know? More like a measure of non-information?Mung
September 5, 2012
September
09
Sep
5
05
2012
05:27 PM
5
05
27
PM
PDT
To elaborate on what Bill said, if we have a fair coin, it can exist in two microstates: heads (call it microstate 1) or tails (call it microstate 2).
I have to disagree with Bill. I have a coin in my pocket and it's not in either the heads state or the tails state.Mung
September 5, 2012
September
09
Sep
5
05
2012
05:00 PM
5
05
00
PM
PDT
1 2 3

Leave a Reply