A Designed Object’s Entropy Must Increase for Its Design Complexity to Increase – Part 1

PS: The most probable or equilibrium cluster of microstates consistent with a given macrostate, is the cluster that has the least information about it, and the most freedom of variation of mass and energy distribution at micro level. This high entropy state-cluster is strongly correlated with high levels of disorder, for reasons connected to the functionality constraints just above. And in fact -- never mind those who are objecting and pretending that this is not so -- it is widely known in physics that entropy is a metric of disorder, some would say it quantifies it and gives it structured physical expression in light of energy and randomness or information gap considerations.kairosfocus

September 8, 2012

September

09

Sep

8

08

2012

12:23 AM

12

12

23

AM

PST

Copy Comment Link

Mung: One more time [cf. 56 above, which clips elsewhere . . . ], let me clip Shannon, 1950/1:

The entropy is a statistical parameter which measures, in a certain sense, how much information is produced on the average for each letter of a text in the language. If the language is translated into binary digits (0 or 1) in the most efficient way, the entropy is the average number of binary digits required per letter of the original language. The redundancy, on the other hand, measures the amount of constraint imposed on a text in the language due to its statistical structure, e.g., in English the high fre-quency of the letter E, the strong tendency of H to follow T or of V to follow Q. It was estimated that when statistical effects extending over not more than eight letters are considered the entropy is roughly 2.3 bits per letter, the redundancy about 50 per cent.

Going back to my longstanding, always linked note, which I have clipped several times over the past few days, here on is how we measure info and avg info per symbol:

To quantify the above definition of what is perhaps best descriptively termed information-carrying capacity, but has long been simply termed information (in the "Shannon sense" - never mind his disclaimers . . .), let us consider a source that emits symbols from a vocabulary: s1,s2, s3, . . . sn, with probabilities p1, p2, p3, . . . pn. That is, in a "typical" long string of symbols, of size M [say this web page], the average number that are some sj, J, will be such that the ratio J/M --> pj, and in the limit attains equality. We term pj the a priori -- before the fact -- probability of symbol sj. Then, when a receiver detects sj, the question arises as to whether this was sent. [That is, the mixing in of noise means that received messages are prone to misidentification.] If on average, sj will be detected correctly a fraction, dj of the time, the a posteriori -- after the fact -- probability of sj is by a similar calculation, dj. So, we now define the information content of symbol sj as, in effect how much it surprises us on average when it shows up in our receiver: I = log [dj/pj], in bits [if the log is base 2, log2] . . . Eqn 1 This immediately means that the question of receiving information arises AFTER an apparent symbol sj has been detected and decoded. That is, the issue of information inherently implies an inference to having received an intentional signal in the face of the possibility that noise could be present. Second, logs are used in the definition of I, as they give an additive property: for, the amount of information in independent signals, si + sj, using the above definition, is such that: I total = Ii + Ij . . . Eqn 2 For example, assume that dj for the moment is 1, i.e. we have a noiseless channel so what is transmitted is just what is received. Then, the information in sj is: I = log [1/pj] = - log pj . . . Eqn 3 This case illustrates the additive property as well, assuming that symbols si and sj are independent. That means that the probability of receiving both messages is the product of the probability of the individual messages (pi *pj); so: Itot = log1/(pi *pj) = [-log pi] + [-log pj] = Ii + Ij . . . Eqn 4 So if there are two symbols, say 1 and 0, and each has probability 0.5, then for each, I is - log [1/2], on a base of 2, which is 1 bit. (If the symbols were not equiprobable, the less probable binary digit-state would convey more than, and the more probable, less than, one bit of information. Moving over to English text, we can easily see that E is as a rule far more probable than X, and that Q is most often followed by U. So, X conveys more information than E, and U conveys very little, though it is useful as redundancy, which gives us a chance to catch errors and fix them: if we see "wueen" it is most likely to have been "queen.") Further to this, we may average the information per symbol in the communication system thusly (giving in termns of -H to make the additive relationships clearer): - H = p1 log p1 + p2 log p2 + . . . + pn log pn or, H = - SUM [pi log pi] . . . Eqn 5 H, the average information per symbol transmitted [usually, measured as: bits/symbol], is often termed the Entropy; first, historically, because it resembles one of the expressions for entropy in statistical thermodynamics. As Connor notes: "it is often referred to as the entropy of the source." [p.81, emphasis added.] Also, while this is a somewhat controversial view in Physics, as is briefly discussed in Appendix 1below, there is in fact an informational interpretation of thermodynamics that shows that informational and thermodynamic entropy can be linked conceptually as well as in mere mathematical form . . .

What this last refers to is the Gibbs formulation of entropy for statistical mechanics, and its implications when the relationship between probability and information is brought to bear in light of the Macro-micro views of a body of matter. That is, when we have a body, we can characterise its state per lab-level thermodynamically significant variables, that are reflective of many possible ultramicroscopic states of constituent particles. Thus, clipping again from my always linked discussion that uses Robertson's Statistical Thermophysics, CH 1 [and do recall my strong recommendation that we all acquire and read L K Nash's elements of Statistical Thermodynamics as introductory reading):

Summarising Harry Robertson's Statistical Thermophysics (Prentice-Hall International, 1993) . . . . For, as he astutely observes on pp. vii - viii:

. . . the standard assertion that molecular chaos exists is nothing more than a poorly disguised admission of ignorance, or lack of detailed information about the dynamic state of a system . . . . If I am able to perceive order, I may be able to use it to extract work from the system, but if I am unaware of internal correlations, I cannot use them for macroscopic dynamical purposes. On this basis, I shall distinguish heat from work, and thermal energy from other forms . . .

And, in more details, (pp. 3 - 6, 7, 36, cf Appendix 1 below for a more detailed development of thermodynamics issues and their tie-in with the inference to design; also see recent ArXiv papers by Duncan and Samura here and here):

. . . It has long been recognized that the assignment of probabilities to a set represents information, and that some probability sets represent more information than others . . . if one of the probabilities say p2 is unity and therefore the others are zero, then we know that the outcome of the experiment . . . will give [event] y2. Thus we have complete information . . . if we have no basis . . . for believing that event yi is more or less likely than any other [we] have the least possible information about the outcome of the experiment . . . . A remarkably simple and clear analysis by Shannon [1948] has provided us with a quantitative measure of the uncertainty, or missing pertinent information, inherent in a set of probabilities [NB: i.e. a probability different from 1 or 0 should be seen as, in part, an index of ignorance] . . . . [deriving informational entropy, cf. discussions here, here, here, here and here; also Sarfati's discussion of debates and the issue of open systems here . . . ] H({pi}) = - C [SUM over i] pi*ln pi, [. . . "my" Eqn 6] [--> This is essentially the same as Gibbs Entropy, once C is properly interpreted and the pi's relate to the probabilities of microstates consistent with the given lab-observable macrostate of a system at a given Temp, with a volume V, under pressure P, degree of magnetisation, etc etc . . . ] [where [SUM over i] pi = 1, and we can define also parameters alpha and beta such that: (1) pi = e^-[alpha + beta*yi]; (2) exp [alpha] = [SUM over i](exp - beta*yi) = Z [Z being in effect the partition function across microstates, the "Holy Grail" of statistical thermodynamics]. . . . [H], called the information entropy, . . . correspond[s] to the thermodynamic entropy [i.e. s, where also it was shown by Boltzmann that s = k ln w], with C = k, the Boltzmann constant, and yi an energy level, usually ei, while [BETA] becomes 1/kT, with T the thermodynamic temperature . . . A thermodynamic system is characterized by a microscopic structure that is not observed in detail . . . We attempt to develop a theoretical description of the macroscopic properties in terms of its underlying microscopic properties, which are not precisely known. We attempt to assign probabilities to the various microscopic states . . . based on a few . . . macroscopic observations that can be related to averages of microscopic parameters. Evidently the problem that we attempt to solve in statistical thermophysics is exactly the one just treated in terms of information theory. It should not be surprising, then, that the uncertainty of information theory becomes a thermodynamic variable when used in proper context . . . . Jayne's [summary rebuttal to a typical objection] is ". . . The entropy of a thermodynamic system is a measure of the degree of ignorance of a person whose sole knowledge about its microstate consists of the values of the macroscopic quantities . . . which define its thermodynamic state. This is a perfectly 'objective' quantity . . . it is a function of [those variables] and does not depend on anybody's personality. There is no reason why it cannot be measured in the laboratory." . . . . [pp. 3 - 6, 7, 36; replacing Robertson's use of S for Informational Entropy with the more standard H.]

As is discussed briefly in Appendix 1, Thaxton, Bradley and Olsen [TBO], following Brillouin et al, in the 1984 foundational work for the modern Design Theory, The Mystery of Life's Origins [TMLO], exploit this information-entropy link, through the idea of moving from a random to a known microscopic configuration in the creation of the bio-functional polymers of life, and then -- again following Brillouin -- identify a quantitative information metric for the information of polymer molecules. For, in moving from a random to a functional molecule, we have in effect an objective, observable increment in information about the molecule. This leads to energy constraints, thence to a calculable concentration of such molecules in suggested, generously "plausible" primordial "soups." In effect, so unfavourable is the resulting thermodynamic balance, that the concentrations of the individual functional molecules in such a prebiotic soup are arguably so small as to be negligibly different from zero on a planet-wide scale. By many orders of magnitude, we don't get to even one molecule each of the required polymers per planet, much less bringing them together in the required proximity for them to work together as the molecular machinery of life. The linked chapter gives the details. More modern analyses [e.g. Trevors and Abel, here and here], however, tend to speak directly in terms of information and probabilities rather than the more arcane world of classical and statistical thermodynamics . . .

Now, of course, as Wiki summarises, the classic formulation of the Gibbs entropy is:

The macroscopic state of the system is defined by a distribution on the microstates that are accessible to a system in the course of its thermal fluctuations. So the entropy is defined over two different levels of description of the given system. The entropy is given by the Gibbs entropy formula, named after J. Willard Gibbs. For a classical system (i.e., a collection of classical particles) with a discrete set of microstates, if E_i is the energy of microstate i, and p_i is its probability that it occurs during the system's fluctuations, then the entropy of the system is: S = -k_B * [sum_i] p_i * ln p_i This definition remains valid even when the system is far away from equilibrium. Other definitions assume that the system is in thermal equilibrium, either as an isolated system, or as a system in exchange with its surroundings. The set of microstates on which the sum is to be done is called a statistical ensemble. Each statistical ensemble (micro-canonical, canonical, grand-canonical, etc.) describes a different configuration of the system's exchanges with the outside, from an isolated system to a system that can exchange one more quantity with a reservoir, like energy, volume or molecules. In every ensemble, the equilibrium configuration of the system is dictated by the maximization of the entropy of the union of the system and its reservoir, according to the second law of thermodynamics (see the statistical mechanics article). Neglecting correlations between the different possible states (or, more generally, neglecting statistical dependencies between states) will lead to an overestimate of the entropy[1]. These correlations occur in systems of interacting particles, that is, in all systems more complex than an ideal gas. This S is almost universally called simply the entropy. It can also be called the statistical entropy or the thermodynamic entropy without changing the meaning. Note the above expression of the statistical entropy is a discretized version of Shannon entropy. The von Neumann entropy formula is an extension of the Gibbs entropy formula to the quantum mechanical case. It has been shown that the Gibb's Entropy is numerically equal to the experimental entropy[2] dS = delta_Q/{T} . . .

Looks to me that this is one time Wiki has it just about dead right. Let's deduce a relationship that shows physical meaning in info terms, where (- log p_i) is an info metric, I-i, here for microstate i, and noting that a sum over i of p_i * log p_i is in effect a frequency/probability weighted average or the expected value of the log p_i expression, and also moving away from natural logs (ln) to generic logs:

S_Gibbs = -k_B * [sum_i] p_i * log p_i But, I_i = - log p_i So, S_Gibbs = k_B * [sum_i] p_i * I-i i.e. S-Gibbs is a constant times the average information required to specify the particular microstate of the system, given its macrostate, the MmIG (macro-micro info gap.

Or, as Wiki also says elsewhere:

At an everyday practical level the links between information entropy and thermodynamic entropy are not close. Physicists and chemists are apt to be more interested in changes in entropy as a system spontaneously evolves away from its initial conditions, in accordance with the second law of thermodynamics, rather than an unchanging probability distribution. And, as the numerical smallness of Boltzmann's constant kB indicates, the changes in S / kB for even minute amounts of substances in chemical and physical processes represent amounts of entropy which are so large as to be right off the scale compared to anything seen in data compression or signal processing. But, at a multidisciplinary level, connections can be made between thermodynamic and informational entropy, although it took many years in the development of the theories of statistical mechanics and information theory to make the relationship fully apparent. In fact, in the view of Jaynes (1957), thermodynamics should be seen as an application of Shannon's information theory: the thermodynamic entropy is interpreted as being an estimate of the amount of further Shannon information needed to define the detailed microscopic state of the system, that remains uncommunicated by a description solely in terms of the macroscopic variables of classical thermodynamics. For example, adding heat to a system increases its thermodynamic entropy because it increases the number of possible microscopic states that it could be in, thus making any complete state description longer. (See article: maximum entropy thermodynamics.[Also,another article remarks: >>in the words of G. N. Lewis writing about chemical entropy in 1930, "Gain in entropy always means loss of information, and nothing more" . . . in the discrete case using base two logarithms, the reduced Gibbs entropy is equal to the minimum number of yes/no questions that need to be answered in order to fully specify the microstate, given that we know the macrostate.>>]) Maxwell's demon can (hypothetically) reduce the thermodynamic entropy of a system by using information about the states of individual molecules; but, as Landauer (from 1961) and co-workers have shown, to function the demon himself must increase thermodynamic entropy in the process, by at least the amount of Shannon information he proposes to first acquire and store; and so the total entropy does not decrease (which resolves the paradox).

So, immediately, the use of "entropy" in the Shannon context, to denote not H but N*H, where N is the number of symbols (thus, step by step states emitting those N symbols involved), is an error of loose reference. Similarly, by exploiting parallels in formulation and insights into the macro-micro distinction in thermodynamics, we can develop a reasonable and empirically supportable physical account of how Shannon information is a component of the Gibbs entropy narrative. Where also Gibbs subsumes the Boltzmann formulation and onward links to the lab-measurable quantity. (Nash has a useful, relatively lucid -- none of this topic is straightforward -- discussion on that.) Going beyond, once the bridge is there between information and entropy, it is there. It is not going away, regardless of how inconvenient it may be to some schools of thought. We can easily see that, for example, information is expressed in the configuration of a string, Z, of elements z1 -z2 . . . zN in accordance with a given protocol of assignment rules and interpretation & action rules etc. Where also, such is WLOG as AutoCAD etc show us that using the nodes and arcs representation and a list of structured strings that record this, essentially any object can be described in terms of a suitably configured string or collection of strings. So now, we can see that string Z (with each zi possibly taking b discrete states) may represent an island of function that expresses functionally specific complex organisation and associated information. Because of specificity to achieve and keep function, leading to a demand for matching, co-ordinated values of zi along the string, that string has relatively few of the N^b possibilities for N elements with b possible states being permissible. We are at isolated islands of specific function i.e cases E from a zone of function T in a space of possibilities W. (BTW, once b^N exceeds 500 bits on the gamut of our solar system, or 1,000 bits on the gamut of our observable cosmos, that brings to bear all the needle in the haystack, monkeys at keyboards analysis that has been repeatedly brought forth to show why FSCO/I is a useful sign of IDOW -- intelligently directed organising work -- as empirically credible cause.) We see then that we have a complex string to deal with, with sharp restrictions on possible configs, that are evident from observable function, relative to the general possibility of W = b^N possibilities. Z is in a highly informational, tightly constrained state that comes form a special zone specifiable on macro-level observable function (without actually observing Z directly). That constraint on degrees of freedom contingent on functional, complex organisation, is tantamount to saying that a highly informational state is a low entropy one, in the Gibbs sense. Going back to the expression, comparatively speaking there is not a lot of MISSING micro-level info to be specified, i.e. simply by knowing the fact of complex specified information-rich function, we know that we are in a highly restricted special Zone T in W. This immediately applies to R/DNA and proteins, which of course use string structures. It also applies tot he complex 3-D arrangement of components in the cell, which are organised in ways that foster function. And of course it applies to the 747 in a flyable condition. Such easily explains why a tornado passing through a junkyard in Seattle will not credibly assemble a 747 from parts it hits, and it explains why the raw energy and forces of the tornado that hits another formerly flyable 747, and tearing it apart, would render its resulting condition much less specified per function, and in fact result in predictable loss of function. We will also see that this analysis assumes the functional possibilities of a mass of Al, but is focussed on the issue of functional config and gives it specific thermodynamics and information theory context. (Where also, algebraic modelling is a valid mathematical analysis.) I trust this proves helpful KFkairosfocus

September 8, 2012

September

09

Sep

8

08

2012

12:18 AM

12

12

18

AM

PST

Copy Comment Link

They were not concerned with conceptual, non-mathematical answers to the question, “What is entropy, really?” that their students occasionally had the courage to ask.

Does Lambert answer that question? What is Entropy, really? So, entropy is the answer to the age-old question, why me?Mung

September 7, 2012

September

09

Sep

7

07

2012

05:52 PM

5

05

52

PM

PST

Copy Comment Link

...and when left to itself it rapidly proceeds to the most probable state.

There, I fixed it fer ya! As a bonus you get the "directionality" of entropy. Ordered and disordered gots nothing to do with it.Mung

September 7, 2012

September

09

Sep

7

07

2012

05:41 PM

5

05

41

PM

PST

Copy Comment Link

Boltzmann

“In order to explain the fact that the calculations based on this assumption [“…that by far the largest number of possible states have the characteristic properties of the Maxwell distribution…”] correspond to actually observable processes, one must assume that an enormously complicated mechanical system represents a good picture of the world, and that all or at least most of the parts of it surrounding us are initially in a very ordered — and therefore very improbable — state. When this is the case, then whenever two of more small parts of it come into interaction with each other, the system formed by these parts is also initially in an ordered state and when left to itself it rapidly proceeds to the disordered most probable state.” (Final paragraph of #87, p. 443.)

That slight, innocent paragraph of a sincere man — but before modern understanding of q(rev)/T via knowledge of molecular behavior (Boltzmann believed that molecules perhaps could occupy only an infinitesimal volume of space), or quantum mechanics, or the Third Law — that paragraph and its similar nearby words are the foundation of all dependence on “entropy is a measure of disorder”. Because of it, uncountable thousands of scientists and non-scientists have spent endless hours in thought and argument involving ‘disorder’and entropy in the past century. Apparently never having read its astonishingly overly-simplistic basis, they believed that somewhere there was some profound base. Somewhere. There isn’t. Boltzmann was the source and no one bothered to challenge him. Why should they? Boltzmann’s concept of entropy change was accepted for a century primarily because skilled physicists and thermodynamicists focused on the fascinating relationships and powerful theoretical and practical conclusions arising from entropy’s relation to the behavior of matter. They were not concerned with conceptual, non-mathematical answers to the question, “What is entropy, really?” that their students occasionally had the courage to ask. Their response, because it was what had been taught to them, was “Learn how to calculate changes in entropy. Then you will understand what entropy ‘really is’.” There is no basis in physical science for interpreting entropy change as involving order and disorder.

scordova

September 7, 2012

September

09

Sep

7

07

2012

07:58 AM

7

07

58

AM

PST

Copy Comment Link

From the OP:

How entropy became equated with disorder, I do not know ...

Arieh Ben-Naim writes: "It should be noted that Boltzmann himself was perhaps the first to use the "disorder" metaphor in his writing:

...are initially in a very ordered - therefore very improbable - state ... when left to itself it rapidly proceeds to the disordered most probable state. - Boltzmann (1964)

You should note that Boltzmann uses the terms "order" and "disorder" as qualitative descriptions of what goes on in the system. When he defines entropy, however, he uses either the number of states or probability. Indeed, there are many examples where the term "disorder" can be applied to describe entropy. For instance, mixing two gases is well described as a process leading to a higher degree of disorder. However, there are many examples for which the disorder metaphor fails."Mung

September 6, 2012

September

09

Sep

6

06

2012

10:11 AM

10

10

11

AM

PST

Copy Comment Link

As I have said above, the adoption of the term "entropy" for SMI was an unfortunate event, not because entropy is not SMI, but because SMI is not entropy!.

*SMI - Shannon's Measure of Information http://www.worldscientific.com/worldscibooks/10.1142/7694Mung

September 6, 2012

September

09

Sep

6

06

2012

09:18 AM

9

09

18

AM

PST

Copy Comment Link

OOPS, 600 + Exa BYTESkairosfocus

September 6, 2012

September

09

Sep

6

06

2012

07:47 AM

7

07

47

AM

PST

Copy Comment Link

PS: As I head out, I think an estimate of what it would take to describe the state of 1 cc of monoatomic ideal gas at 760 mm HG and 0 degrees C, i.e. 2.687 * 10^19 particles with 6 degrees of positional and momentum freedom would help us. Let us devote 32 bits -- 16 bits to get 4 hex sig figs, and a sign bit plus 15 bits for the binary exponent to each of the (x, y, z) and (P_x, P-y and P-z) co-ordinates in the phase space. We are talking about:

2.687 * 10^19 particles x 32 bits per degree of freedom x 6 degrees of freedom each _____________ 5.159 * 10^21 bits of info

That is, to describe the state of the system at a given instant, we would need 5.159 * 10^21 bits, or 644.9 * 10^18 bits. That is how many yes/no quest5ions, in teh correct order, would have to be amnswered and processed every clock tick we update. And with 10^-14 s as a reasonable chemical reaction rate, we are seeing a huge amount of required processing to keep track. As to how that would be done, that is anybody's guess.kairosfocus

September 6, 2012

September

09

Sep

6

06

2012

07:46 AM

7

07

46

AM

PST

Copy Comment Link

F/N: This from OP needs comment:

what is the Shannon Entropy of a system of 500 distinct coins? Answer: 500 bits, or the Universal Probability Bound. By way of extension, if we wanted to build an operating system like Windows-7 that requires gigabits of storage, we would require the computer memory to contain gigabits of Shannon entropy. This illustrates the principle that more complex designs require larger Shannon entropy to support the design. It cannot be otherwise. Design requires the presence of entropy, not absence of it.

Actually, in basic info theory, H strictly is a measure of average info content per element in a system or symbol in a message. Hence its being estimated on a weighted average of information per relevant element. This, I illustrated earlier from a Shannon 1950/1 paper, in comment 15 in the part 2 thread:

The entropy is a statistical parameter which measures, in a certain sense, how much information is produced on the average for each letter of a text in the language. If the language is translated into binary digits (0 or 1) in the most efficient way, the entropy is the average number of binary digits required per letter of the original language. The redundancy, on the other hand, measures the amount of constraint imposed on a text in the language due to its statistical structure, e.g., in English the high fre-quency of the letter E, the strong tendency of H to follow T or of V to follow Q. It was estimated that when statistical effects extending over not more than eight letters are considered the entropy is roughly 2.3 bits per letter, the redundancy about 50 per cent.

So, we see the context of usage here. But what happens when you have a message of N elements? In the case of a system of complexity N elements, then the cumulative, Shannon metric based information -- notice how I am shifting terms to avoid ambiguity -- is, logically, H + H + . . . H N times over, or N * H. And, as was repeatedly highlighted, in the case of the entropy of systems that are in clusters of microstates consistent with a macrostate, the thermodynamic entropy is usefully measured by and understood on terms of the Macro-micro information gap (MmIG], not on a per state or per particle basis but a cumulative basis: we know macro quantities, not the specific position and momentum of each particle, from moment to moment, which given chaos theory we could not keep track of anyway. A useful estimate per the Gibbs weighed probability sum entropy metric -- which is where Shannon reputedly got the term he used from in the first place, on a suggestion from von Neumann -- is:

>>in the words of G. N. Lewis writing about chemical entropy in 1930, "Gain in entropy always means loss of information, and nothing more" . . . in the discrete case using base two logarithms, the reduced Gibbs entropy is equal to the minimum number of yes/no questions that need to be answered in order to fully specify the microstate, given that we know the macrostate. >>

Where, Wiki gives a useful summary:

The macroscopic state of the system is defined by a distribution on the microstates that are accessible to a system in the course of its thermal fluctuations. So the entropy is defined over two different levels of description of the given system. The entropy is given by the Gibbs entropy formula, named after J. Willard Gibbs. For a classical system (i.e., a collection of classical particles) with a discrete set of microstates, if E_i is the energy of microstate i [--> Notice, summation is going to be over MICROSTATES . . . ], and p_i is its probability that it occurs during the system's fluctuations, then the entropy of the system is S_sys = - k_B [SUM over i's] P_i log p_i

Also, {- log p_i} is an information metric, I_i, i.e the information we would learn on actually coming to know that the system is in microstate i. Thus, we are taking a scaled info metric on the probabilistically weighted summmation of info in each microstate. Let us adjust: S_sys = k_B [SUM over i's] p_i * I_i This is the weighted average info per possible microstate, scaled by k_B. (Which of course is where the Joules per Kelvin come from.) In effect the system is giving us a message, its macrostate, but that message is ambiguous over the specific microstate in it. After a bit of mathematical huffing and puffing, we are seeing that the entropy is linked to the average info per possible microstate. Where this is going is of course that when a system is in a state with many possible microstates, it has enormous freedom of being in possible configs, but if the macro signals lock us down to specific states in small clusters, we need to account for how it could be in such clusters, when under reasonable conditions and circumstances, it could be easily in states that are far less specific. In turn that raises issues over IDOW. Which then points onward to FSCO/I being a sign of intelligent design. KFkairosfocus

September 6, 2012

September

09

Sep

6

06

2012

07:18 AM

7

07

18

AM

PST

Copy Comment Link

But what if you only have the encoded string to work upon, and the JPEG codec generates an apparently random string as output? How do you tell whether the output signal is truly random or that it contains a human-readable message encoded using some other protocol?

You can't tell if a string is truly the product of mindless purposeless forces (random is your word), so you have to be agnostic about that. So one must accept that one can make a false inference to randomness (such as when someone wants to be extremely stealthy and encrypt the data). If it parses with another codec that is avaiable to you, you have good reason to accept the file is designed. Beyond that, one might have other techniques such as those that team Norton Symantec used to determine that Stuxnet was the product of an incredible level of intelligent design: How Digital Detectives Deciphered Stuxnet

Several layers of masking obscured the zero-day exploit inside, requiring work to reach it, and the malware was huge — 500k bytes, as opposed to the usual 10k to 15k. Generally malware this large contained a space-hogging image file, such as a fake online banking page that popped up on infected computers to trick users into revealing their banking login credentials. But there was no image in Stuxnet, and no extraneous fat either. The code appeared to be a dense and efficient orchestra of data and commands. .... Instead, Stuxnet stored its decrypted malicious DLL file only in memory as a kind of virtual file with a specially crafted name. It then reprogrammed the Windows API — the interface between the operating system and the programs that run on top of it — so that every time a program tried to load a function from a library with that specially crafted name, it would pull it from memory instead of the hard drive. Stuxnet was essentially creating an entirely new breed of ghost file that would not be stored on the hard drive at all, and hence would be almost impossible to find. O Murchu had never seen this technique in all his years of analyzing malware. “Even the complex threats that we see, the advanced threats we see, don’t do this,” he mused during a recent interview at Symantec’s office. Clues were piling up that Stuxnet was highly professional, and O Murchu had only examined the first 5k of the 500k code. It was clear it was going to take a team to tackle it. The question was, should they tackle it? .... But Symantec felt an obligation to solve the Stuxnet riddle for its customers. More than this, the code just seemed way too complex and sophisticated for mere espionage. It was a huge adrenaline-rush of a puzzle, and O Murchu wanted to crack it. “Everything in it just made your hair stand up and go, this is something we need to look into,” he said. .... As Chien and O Murchu mapped the geographical location of the infections, a strange pattern emerged. Out of the initial 38,000 infections, about 22,000 were in Iran. Indonesia was a distant second, with about 6,700 infections, followed by India with about 3,700 infections. The United States had fewer than 400. Only a small number of machines had Siemens Step 7 software installed – just 217 machines reporting in from Iran and 16 in the United States. The infection numbers were way out of sync with previous patterns of worldwide infections — such as what occurred with the prolific Conficker worm — in which Iran never placed high, if at all, in infection stats. South Korea and the United States were always at the top of charts in massive outbreaks, which wasn’t a surprise since they had the highest numbers of internet users. But even in outbreaks centered in the Middle East or Central Asia, Iran never figured high in the numbers. It was clear the Islamic Republic was at the center of the Stuxnet infection. The sophistication of the code, plus the fraudulent certificates, and now Iran at the center of the fallout made it look like Stuxnet could be the work of a government cyberarmy — maybe even a United States cyberarmy.

And that illustrates how a non-random string in a computer might be deduced as the product of some serious ID.scordova

September 6, 2012

September

09

Sep

6

06

2012

07:00 AM

7

07

00

AM

PST

Copy Comment Link

But even if you didn’t know in advance, the fact that a JPEG decoder could produce a meaningful image proves only that the message was encoded using the JPEG protocol.

And JPEG encoders are intellignetly deisgned, so the files generated are still products of intelligent design.

A magnificent feat of inference.

Indeed.

It might be interesting if you could prove that the message originated from a non-human source

Humans can make JPEGs, so no need to invoke non-human sources.scordova

September 6, 2012

September

09

Sep

6

06

2012

06:32 AM

6

06

32

AM

PST

Copy Comment Link

TA: Why not look over in the next thread 23 - 24 (with 16 in context as background)? Kindly explain the behaviour of the black box that emits ordered vs random vs meaningful text strings of 502 bits: || BLACK BOX || –> 502 bit string As in, explain to us, how emitting the string of ASCII characters for the first 72 or so letters of this post is not an excellent reason to infer to design as the material cause of the organised string. As in, intelligently directed organising work, which I will label for convenience, IDOW. Can you justify a claim that lucky noise plus mechanical necessity adequately explains such an intelligible string, in the teeth of what sampling theory tells us on the likely outcome of samples on the gamut of the 10^57 atoms of the solar system for 10^17 s, at about 10^14 sa/s -- comparable to fast chemical ionic reaction rates -- relative to the space of possible configs of 500 bits. (As in 1 straw-size to a cubical hay bale of 1,000 LY on the side about as thick as our galaxy.) As in, we have reason to infer on FSCO/I as an empirically reliable sign of design, no great surprise, never mind your recirculation of long since cogently answered objections. (NB: This is what often happens when a single topic gets split up by using rapid succession of threads with comments. That is why I posted a reference thread, with a link back and no comments.) KFkairosfocus

September 6, 2012

September

09

Sep

6

06

2012

05:45 AM

5

05

45

AM

PST

Copy Comment Link

No timothya- I don't think so. It is obvious. And not one of those biologists can produce any evidence that demonstrates otherwise.Joe

September 6, 2012

September

09

Sep

6

06

2012

05:33 AM

5

05

33

AM

PST

Copy Comment Link

Joe I am clear that you think so. You are in disagreement with almost every practising biologist in the world of science. But that is your choice. In the meantime, can we focus on Sal's proposal?timothya

September 6, 2012

September

09

Sep

6

06

2012

05:24 AM

5

05

24

AM

PST

Copy Comment Link

timothya- there isn't any evidence that natural selection is non-random- just so that we are clear.Joe

September 6, 2012

September

09

Sep

6

06

2012

05:20 AM

5

05

20

AM

PST

Copy Comment Link

Joe posted this:

And timothya- I am still waiting for evidence that natural selection is non-random….

As far as it matters, you have already had your answer in a different thread. This thread seems to be focussed on the "how to identify designedness", so perhaps we should stick to that subject.timothya

September 6, 2012

September

09

Sep

6

06

2012

05:15 AM

5

05

15

AM

PST

Copy Comment Link

Joe posted this:

I’m saying that if you find a file on a compuetr then it a given some agency put it there.

Brilliant insight. Users of computers generate artefacts that are stored in a form determined by the operating system of the computer that they are using (in turn determined by the human designers of the operating system involved). I would be a little surprised if it proved to be otherwise. However, the reliable transformation of input data to stored data in computer storage doesn't help Sal with his problem of how to assign "designedness" to an arbitrary string of input data. He has to show that there is a reliable way to distinguish between a genuinely random string and a pseudorandom string that is hiding a human-readable message, when all he has to go on is the string itself, with no prior knowledge. If he has such a method, I would be fascinated to know what it is.timothya

September 6, 2012

September

09

Sep

6

06

2012

05:06 AM

5

05

06

AM

PST

Copy Comment Link

And timothya- I am still waiting for evidence that natural selection is non-random....Joe

September 6, 2012

September

09

Sep

6

06

2012

04:51 AM

4

04

51

AM

PST

Copy Comment Link

I'm saying that if you find a file on a compuetr then it a given some agency put it there.Joe

September 6, 2012

September

09

Sep

6

06

2012

04:50 AM

4

04

50

AM

PST

Copy Comment Link

Joe posted this:

I would bet that both strings are the product of agency involvement as blind and undirected processes cannot construct a file.

Forget the container and consider the thing contained (I mean, really, do I have to define every parameter of the discussion?). Scientists sensing signals from a pulsar store the results in a computer "file" via a series of truth-preserving transformations (light data to electronics to magnetic marks on a hard drive). Are you arguing that the stored data does not correlate reliably to the original sense data?timothya

September 6, 2012

September

09

Sep

6

06

2012

04:44 AM

4

04

44

AM

PST

Copy Comment Link

Waiting for Sal's response, I noticed that he posted this:

The fact that I knew File C was a JPEG suggests that I had some advanced knowledge of the file being designed. And even if I didn’t know that in advance, the fact that it could be parsed and processed as a JPEG indicates that it is organized.

Exactly. You knew in advance that the file was JPEG-encoded. But even if you didn't know in advance, the fact that a JPEG decoder could produce a meaningful image proves only that the message was encoded using the JPEG protocol. A magnificent feat of inference. It might be interesting if you could prove that the message originated from a non-human source. Otherwise not. But what if you only have the encoded string to work upon, and the JPEG codec generates an apparently random string as output? How do you tell whether the output signal is truly random or that it contains a human-readable message encoded using some other protocol? If I understand your original post, you claim that design is detectable from the pattern of the encoded message, independent of its mode of encoding.timothya

September 6, 2012

September

09

Sep

6

06

2012

04:34 AM

4

04

34

AM

PST

Copy Comment Link

If you have a means of distinguishing between File X, (which contains a genuine random strong), and File Y (which contains a pseudorandom random string encoding a human-readable sentence), then fill your boots and publish the method.

I would bet that both strings are the product of agency involvement as blind and undirected processes cannot construct a file.Joe

September 6, 2012

September

09

Sep

6

06

2012

04:18 AM

4

04

18

AM

PST

Copy Comment Link

If you have a means of distinguishing between File X, (which contains a genuine random strong), and File Y (which contains a pseudorandom random string encoding a human-readable sentence), then fill your boots and publish the method. The sound you can hear is that of computer security specialists the world over shifting uncomfortably in their seats. Or perhaps of computer security specialists laughing their faces off. The point is this: if you want to infer "design" solely from the evidence (of the contents of the files, with no a priori knowledge of their provenance), then what is your method?timothya

September 6, 2012

September

09

Sep

6

06

2012

03:12 AM

3

03

12

AM

PST

Copy Comment Link

If you didn’t know in advance what the origin of File A and File C were, then you would have no useful evidence from the contents of the two files to decide that one was “highly disorganised” and the other was “highly organised”.

The fact that I knew File C was a JPEG suggests that I had some advanced knowledge of the file being designed. And even if I didn't know that in advance, the fact that it could be parsed and processed as a JPEG indicates that it is organized. The fact that I specified in advance that FILE A was created by a random number generator ensures a high probability it will not be designed. File B had to be restated with qualification as gpuccio pointed out. The inference of design or lack thereof was based on advanced prior knowledge, not some explantory filter after the fact.scordova

September 6, 2012

September

09

Sep

6

06

2012

02:30 AM

2

02

30

AM

PST

Copy Comment Link

Correction, I posted this:

Your argument appears to say that if a system transmits a constant signal, then it must be organised.

I meant to use the term from your post that the valid inference for File B was that the file contents were designed. Clearly a gigabit of "ones" is organised in the sense that it has an evident pattern.timothya

September 6, 2012

September

09

Sep

6

06

2012

02:30 AM

2

02

30

AM

PST

Copy Comment Link

Sal posted this:

Now to help disentangle concepts a little further consider three 3 computer files:

File_A : 1 gigabit of binary numbers randomly generated File_B : 1 gigabit of all 1?s File_C : 1 gigabit encrypted JPEG Here are the characteristics of each file: File_A : 1 gigabit of binary numbers randomly generated Shannon Entropy: 1 gigabit Algorithmic Entropy (Kolmogorov Complexity): high Thermodynamic Entropy: N/A Organizational characteristics: highly disorganized inference : not designed File_B : 1 gigabit of all 1?s Shannon Entropy: 1 gigabit Algorithmic Entropy (Kolmogorov Complexity): low Thermodynamic Entropy: N/A Organizational characteristics: highly organized inference : designed (with qualification, see note below) File_C : 1 gigabit encrypted JPEG Shannon Entropy: 1 gigabit Algorithmic Entropy (Kolmogorov complexity): high Thermodynamic Entropy: N/A Organizational characteristics: highly organized inference : extremely designed

Please tell me that you are joking. If you didn't know in advance what the origin of File A and File C were, then you would have no useful evidence from the contents of the two files to decide that one was "highly disorganised" and the other was "highly organised". Hint: the purpose of encryption is to make the contents of the file approach as closely as possible to a randomly generated string. File B supports an inference of "highly organised"? How? Why? What if the ground state of the signal is just the continuous emission of something interpreted digitally as "ones" (or zeroes" for that matter). Your argument appears to say that if a system transmits a constant signal, then it must be organised.timothya

September 6, 2012

September

09

Sep

6

06

2012

02:20 AM

2

02

20

AM

PST

Copy Comment Link

FROM MUNG: No Sal, 500 pennies gets you 500 bits of copper plated zinc, not 500 bits of information (or Shannon entropy).

Contrast to Bill Dembski's recent article:

FROM BILL DEBMSKI In the information-theory literature, information is usually characterized as the negative logarithm to the base two of a probability (or some logarithmic average of probabilities, often referred to as entropy). This has the effect of transforming probabilities into bits and of allowing them to be added (like money) rather than multiplied (like probabilities). Thus, a probability of one-eighths, which corresponds to tossing three heads in a row with a fair coin, corresponds to three bits,

I just did a comparable calculation more elaborately, and you missed it. Instead of tossing a single coin 3 times, I had 3 coins tossed 1 time.

FROM WIKI A single toss of a fair coin has an entropy of one bit. A series of two fair coin tosses has an entropy of two bits. The entropy rate for the coin is one bit per toss

I wrote the analogous situation, except insted of making multiple tosses of a single coin, I did the formula for single tosses of multiple coins. The Shannon entropy is analogous.

I wrote: It can be shown that for the Shannon entropy of a system of N distinct coins is equal to N bits. That is, a system with 1 coin has 1 bit of Shannon entropy, a system with 2 coins has 2 bits of Shannon entropy, a system of 3 coins has 3 bits of Shannon entropy, etc.

scordova

September 5, 2012

September

09

Sep

5

05

2012

10:18 PM

10

10

18

PM

PST

Copy Comment Link

Entropy:

The notion of entropy is foundational to physics, engineering, information theory and ID. These essays are written to provide a discussion on the topic of entropy and its relationship to other concepts such as uncertainty, probability, microstates, and disorder. Much of what is said will go against popular understanding, but the aim is to make these topics clearer.

ok, so what is entropy?

First I begin with calculating Shannon entropy for simple cases.

ok, but first, what is "Shannon entropy"?

2. Shannon entropy – measured in bits or dimensionless units

Telling me it's measured in bits doesn't tell me what "it" is.

I is the Shannon entropy (or measure of information).

So "Shannon entropy" is a measure of information?

Hence, sometimes entropy is described in terms of improbability or uncertainty or unpredictability.

So Shannon entropy is a measure of what we don't know? More like a measure of non-information?Mung

September 5, 2012

September

09

Sep

5

05

2012

05:27 PM

5

05

27

PM

PST

Copy Comment Link

To elaborate on what Bill said, if we have a fair coin, it can exist in two microstates: heads (call it microstate 1) or tails (call it microstate 2).

I have to disagree with Bill. I have a coin in my pocket and it's not in either the heads state or the tails state.Mung

September 5, 2012

September

09

Sep

5

05

2012

05:00 PM

5

05

00

PM

PST

Copy Comment Link