Biologist Wayne Rossiter on Joshua Swamidass’ claim that entropy = information

Thanks, guppio. No problem. Looks like you've got an awesome OP up, and I look forward to checking it out. You don't have to agree with me on all the entropy details. Very good exchange anyway and it has been useful. In any event, we certainly agree on the larger implications. I'm just hoping something I wrote may pique your interest or cause any who may stop by and read this exchange to sit up and say, "Wait a minute, maybe I need to think through this again in more detail . . ." That is all I can hope for. :) Eric Anderson

Eric Anderson: Thank you for the further comments. Sorry if I am late in answering, but I have been busy with a new OP in the last day. I don't know what to say. I still think the the same concepts that are valid for informational entropy in strings do apply to thermodynamic entropy in physical systems, but as I have said I am not a physicist and therefore I cannot really explain the correspondence. Maybe KF could help here. I see that you are not comfortable with this idea, but perhaps we should deepen our understanding of thermodynamics and of the probabilistic approach to the second law. However, for informational entropy I think we have reached a very good understanding. It's always a pleasure to have a good exchange with you. :) gpuccio

gpuccio @50: Returning to the other part of your comment:

The point is: most random configurations, both for strings and for physical systems, are different . . .

Yes, random configurations are different, by virtue of their randomness. But what is it that we think we are trying to describe in a physical system?

Ordered states are compressible, and usually have a vey specific macro-behaviour. But their entropy is low. Moreover, they are exceedingly rare.

Ordered states, in the sense of functional organization are exceedingly rare. But simple order is not so rare. Crystals, waves, snowflakes -- many things that are driven by law-like processes tend toward mere "order" (and, therefore, compressibility in their description).

Random states . . . are all different, and scarcely compressible. Therefore, they have high entropy. But their macro-behaviour is strikingly similar (the “homogeneity” of which you speak).

And thus we return to one of the key questions, but I like the words you've used, so let me reframe it: Why do we think we get to describe an ordered state in macro-behavior terms, but we have to describe a random state in micro-behavior terms? I could just as well decide I want to do the opposite. In which case my brief description of the random system at the macro level would be highly compressible, while my lengthy and detailed description of the ordered (at least a functionally organized) system at a micro level would not. The point is, we are making a decision about what to describe. This brings me back to a fundamental point. There is no information in physical things by their mere existence. Even if we were to grant the (somewhat questionable, it seems to me) idea that entropy drives toward random states and, further that for some reason we have to describe random states in micro terms -- even if we grant all that, it doesn't mean there is any information contained in the system. All it means is that it requires more information for us to describe the system, to whatever level of detail we have decided is sufficient for our purposes. Eric Anderson

gpuccio: Thanks for your comments. I am fully on board with your ultimate conclusion:

Swamidass is talking of entropy, not of information. But he is trying to suggest that entropy means information. And that is not true.

And also agree with the general thrust of how you view things. I appreciate you sticking with me this long to delve into the nuances! --- Just one quick observation this evening, and then I'll hopefully have a chance to respond with more detail tomorrow. The problem with strings is that we often view an entropic string (random string) as being made up of a sequence of various characters. However, in these kinds of discussions everyone almost always ignores how or what made the changes to the string. So in a typical example we start with a sense string (say, made up from the 26 characters of the English alphabet) and then introduce (who introduces? what introduces?) random characters to the string. Sure enough, the string ends up less ordered and less compressible than when we started. Entropy in the real world doesn't act that way, it seems. Entropy doesn't randomly drive particles toward one of 26 possible states. It isn't "randomly" picking states and assigning them to particles, the way our random characters get jumbled in a string. If we are going to talk about strings, a more apt analogy to the physical world would be if we were to start with a sense string and then introduce a bunch of "A" characters over time. Nothing but "A". Over and over. Always driving toward the same simple state. Or, if we prefer a digital example, let's say we have our files stored on a hard drive that, unfortunately, over time replaces bits with a zero. We sometimes think of entropy as being random. But it is "random" only in the sense that it is stochastic and may or may not impact a particular particle at a given instant. But it is not at all "random" in the sense of where it is driving. It isn't "randomly" deciding whether to drive toward a bunch of different energy states (like our random character intrusions into a string). As a lawful process, it always drives in the same direction. So it seems to me there is a basic disconnect between the idea of how we get a random string and how entropy drives toward a particular state in the physical world. We could no doubt come up with some real-world examples in which entropy would -- temporarily as it moves away from complex functional organization -- result in a more heterogeneous state. But ultimately it seems it is inexorably driving in a single direction. It drives toward homogeneity, toward sameness, toward blandness, toward the "heat death of the universe", whatever we want to call it. But ultimately it is driving downhill -- always in at least the same general direction. Eric Anderson

Eric Anderson: Well, I am not a physicist, and I usually avoid to deal with problems regarding the second law. I can only say that, intuitively, I think that in essence there is no difference between the concept of Shannon entropy for strings and the concept of entropy in physical systems. I believe that such a think has been demonstrated, too, but probably KF could say something more specific about that. His quote at #15 can certainly be of help. I will try to explain how I see things, very simply. You say: "Now, I ask: Q: In the three-dimensional, physical world, does entropy generally drive toward homogeneity/uniformity or toward heterogeneity? A: It drives toward homogeneity/uniformity." I am not sure that is correct. To understand what I mean, let's start with strings. Let's imagine a long string, say 600 characters. Each character is a value of a three letter alphabet, R, G and B. Therefore, the potential maximum information in such a string is 3^600, IOWs about 951 bits. Now, let's imagine that the string is in reality a sequence of pixels on a monitor, and the three letters are really the basic colors in the RGB system. That carries us immediately from the realm of pure information to the realm of physical events. Now, let's start with a highly ordered sequence: 200 R, 200 G, 200 B. In an informational sense, we see a line made of three parts, each of a different color. That's a very ordered state. From our discussion about strings, we know that it is also highly compressible. Moreover, it is exceedingly rare: only one sequence corresponds to such an ordered state, out of the 3^900 possible sequences. Now, let's say that instead we start from a random sequence of the three letters. What will we see? Most probably, a line in some shade of grey. Not white, because white is a very ordered state too: the most similar thing to pure white, I believe, should be a regular sequence of RGBRGBRGB and so on. Now, let's say we start from a completely different random sequence. It is completely different from the first random one, and if we want to specify it instead of the first random one, we really need to give all the specific values of the sequence. IOWs, both the first random sequence and the second one are different and scarcely compressible, like most random sequences. So, what do we see? With great probability, a very similar line in a similar shade of grey. The point is: most random configurations, both for strings and for physical systems, are different, scarcely compressible and therefore have high entropy, be it Shannon entropy or thermodynamic entropy. Ordered states are compressible, and usually have a vey specific macro-behaviour. But their entropy is low. Moreover, they are exceedingly rare. Random states, instead, constitute the bulk of configurations. Those configurations are all different, and scarcely compressible. Therefore, they have high entropy. But their macro-behaviour is strikingly similar (the "homogeneity" of which you speak). That's how I see the problem. But, again, I am not an expert of the second law and its implications. The single important point is: entropy, be it Shannon's or thermodynamic, has nothing to do with information as we use the word in common language. The meaning of the word "information" in common language corresponds perfectly to the more rigorous concept of "specified information" or "functional information". Swamidass is talking of entropy, not of information. But he is trying to suggest that entropy means information. And that is not true. gpuccio

gpuccio: There are a few more detailed nuances we can discuss, but I wanted to circle back to a basic starting point: We've established that a more repetitive (more homogeneous, less heterogeneous) string is, in principle and under basic algorithms, more compressible. Now, I ask: Q: In the three-dimensional, physical world, does entropy generally drive toward homogeneity/uniformity or toward heterogeneity? A: It drives toward homogeneity/uniformity. What, then, is the basis for the claim that entropy produces more information content? ----- Even setting aside Dr. Swamidass' complete failure of understanding in referring to "information = entropy", what is it that makes someone think entropy will drive toward a less compressible description (given that he also seems stuck in the Shannon bit-calculation mistake, as far as biological information is concerned)? Eric Anderson

Eric Anderson: Just a comment, for the moment. If the system is set in a very general way, we can certainly think of it as receiving and executing automatically both messages and algorithms. That's what I meant when I made a reference to a Turing machine. Think of the receiver as a universal Turing machine that can automatically receive strings, and treat them as pure outcomes or as computational algorithms that generate the outcome. A universal Turing machine is a very useful abstraction when we reason about information problems. So, in this case, transmitting the algorithm to compute pi digits (or any other outcome that is computable) will be in principle the same thing as transmitting an outcome as it is. In both cases, we send a series of bits. The algorithm is just a series of bits that, when received by the universal Turing machine, performs a computation and then generates the outcome and stops. So, in this context, transmitting the compression algorithm is not different from transmitting an outcome. No special understanding is required on the part of the receiver, which works automatically, except for the initial setting of the communication protocol and of the universal Turing machine, which is implemented so that it can receive and execute any kind of algorithm. I say this because, if we want to reason in terms of Shannon entropy and Kolmogorov complexity, we must set the system in a certain way, so that our results are really independent from any added knowledge, except for the initial setting. I am looking forward to your further considerations! gpuccio

Origenes: I agree with what you say. I think we are saying similar things, it's really a question of words and definitions. Maybe I can explain it better this way: You speak of function as indicating the conscious purpose of the designer. That is perfectly fine for me. I have always stressed the role of consciousness in design: design derives from a conscious representation, and that conscious representation implies both cognition (the understanding) and purpose (a feeling connected to the meaning of the desired outcome). I have also always said that conscious experiences cannot ever be defined in an outward, purely objective way. "Purpose" cannot be defined as an objective concept, because it arises in feeling. Even "meaning" has no objective definition, independent from the conscious intuition of a subject. That said, you will probably understand better what I mean with "function" as the basis for design inference if you think of it not as "purpose", but rather as a "functionality" in the object. The functionality is a specific configuration which has been given to the object by the designer, in a context which depended on his conscious view of the whole and on his purpose. However, the specific configuration which makes the object functional so that it can do what the designer wanted to get from that specific object, well, that configuration is objectively in the object. Once it has been outputted by the designer to the object, it is there. It is "inherent" to the object, whether it is in the right context or not. A watch remains a watch, even if we find it on Mars, where nobody is using it to measure time. Why? Because we can use it to measure time, if we want. And anybody else can do the same. So, an enzyme designed to do a specific thing, designed in a context and with an awareness of the whole, still retains its functionality even if isolated from the whole. Why is this important? Because that objective functionality, even if out of context, is enough to infer design, if it is complex enough: IOWs, if a great number of specific bits of configuration are objectively necessary to make an object capable to implement that functionality. That's what I mean by "local function": just the functionality that has been given to the object by the designer. And the linked complexity allows the design inference from that object, because, while simple functionalities can be found in non designed objects (like the stone that can be used as a paper-weight), complex functionalities are never found if not in designed objects (a computer, a watch, an enzyme, and so on). I hope that we can agree on this approach, which in no way denies the truth of all the things that you have said. Thank you anyway for your deep contribution. :) gpuccio

gpuccio @40: Thank you for the fulsome and helpful response. I agree that if: 1. We have a string that is somewhat repetitive, 2. The string is sufficiently long, and 3. We have a compression algorithm based on identifying and re-coding repetitive elements (rather than a dictionary-based algorithm), Then, a. We can create a compressed communication that is shorter than the original string itself, b. We can compress the string more than we could a less repetitive string, and c. The algorithm should also work in general for any string we throw at it (as long as it can recognize the characters). In that sense I agree that we can, and should, view certain strings as being more compressible than others, as a result of their more repetitive properties. ----- About my random string example, you said:

That’s OK, but obviously it is longer than simply transmitting the bits of the string itself, because those bits have however to be included in the algorithm.

Exactly. That is part of the point I have been making all along. In this kind of situation, the information involved in the setup and preparation of the compressed message -- what has to be communicated between sender and receiver -- will always be longer and include more complex specified information than the string itself. This is because in assigning an additional code on top of the string, we are adding additional information. The savings comes in when we have a sufficiently long message or many messages over time. This is the way any dictionary-based compression has to work, such as my example. Also, even a repetitive-analysis compression that could be used on any string works this way: it has to identify the repetitive element, then assign an additional coded element to that repetitive element, then transmit both the repetitive element and the additional coded element to the recipient. The initial process is always longer than just transmitting the repetitive element. The savings comes in when there is enough repetition -- either in a sufficiently large string, or in additional communications over time. ----- Now, there is another kind of "compression" possible, beyond dictionary-based or repetition-based compression. This is what you are referring to in your response to my pi example. You mentioned that for pi:

In alternative, we can simply transmit an algorithm which, according to the basic rules of the system, will generate the string as soon as received . . .

It is true that for a subset of communications we can recognize we are dealing with a mathematical calculation and, if the result of the calculation is lengthy enough, there will be a point at which communicating the calculation instruction is shorter than communicating the result. Note, however, that even in these cases we are piggybacking on a tremendous amount of background knowledge that has already been communicated to the recipient (either by us or by someone else previously), namely, what math is, how it works, what the various symbols represent, and how to calculate pi. Now, for the rare case of an infinite or near-infinite result (such as pi) the aggregate amount of information required to compute pi will end up being less than the string itself. But this case isn't so far removed from the other compression approaches I mentioned before. There is a large amount of information required up front -- more than a reasonable-length string itself. However, for a sufficiently lengthy string or for situations in which we are repeating an operation over time, we have a savings. I should add that a mathematical operation compression approach is, at some level, no different than a dictionary-based approach. In other words, we are taking advantage of the background knowledge and domain expertise of the recipient to generate the string on the receiving end. It is not that we have transmitted the string, or even that we have really transmitted the actual instructions on how to calculate the string. Rather, we have transmitted a message that someone with the correct and sufficient background information will know what to do with it. It is not completely clear to me that such situations are a legitimate form of "compression", so much as a re-coding of the string before transmission. A non-mathematical example might be helpful. We could transmit the message "write the Gettysburg Address." If the recipient has the correct and sufficient background knowledge they can re-create the string. But not because we have actually "compressed" the string. Rather, we have relied on the shared background knowledge of sender and recipient and have, instead of compressing the original string, re-coded the message to achieve the same result and then transmitted that. Telling someone to use their background knowledge to compute pi is really no different. ----- I'll have to think more about your Kolmogorov comments at the end. My hunch is you may be on to something important, but I need to think through it and understand it a bit better. ----- Beyond that, I think we've largely covered the matter of strings. However, if you'll indulge this thread for a couple more days, I'd like to return to where we started -- entropy in real physical systems, which seems to be where Swamidass' misunderstanding starts . . . Eric Anderson

Origines @44: Good thoughts and examples. I don't disagree with your general thrust. I would just note that it is possible to ascertain what a part is doing, even if we don't have a top-down picture of the whole. We don't need to know everything at the highest level in order to discover and understand something at a lower level. As mentioned, I believe you are focusing on purpose, which is fine. It is also common usage and acceptable to refer to a local function in isolation. If we want to call it "activity" that is fine too. In any event, it sounds like we are all largely in agreement. Just a definitional question. Eric Anderson

GPuccio #29, Thank you for your response. What follows are some general thoughts on ‘function’, ‘wholes’ and materialism. I will try to address the specific points you make later.

Eric Anderson: I think gpuccio is using both terms “function” and “system” in the standard scientific understanding of the words.

Yes and it is a good thing that GPuccio adapts, otherwise he would not be able to step in and destroy their arguments as he so frequently and convincingly does. Nevertheless, I’m of the opinion that there is something terribly wrong with this standard scientific use of those terms. Again, “random system” contains two contradictory terms and so does “local function”. IMO these terms are in place to accommodate materialistic notions and they (the materialists) are robbing us blind. Materialists are hell bound to deny the existence of wholes over and beyond the physical micro-level. But when one denies the whole one forfeits the right to use the word ‘function.’ You can’t have your cake and eat it. When one speaks of ‘function’, what one is saying is this: I see in/through/behind all these isolated parts a functionally coherent whole. On my screen I don’t just see isolated random letters, but, instead, I see a coherent whole/message. I see one thing — instead of many things. IOWs the letters you are reading are aspects of one thing. They are functionally integrated in order to convey one specific message. The letters are forced into functional subservience of a message. Notice that we can easily discern a hierarchical structure in this whole: each letter in this post is subservient to words, and the words, in turn, are subservient to sentences, which, in turn … and so forth. Further we may notice that, the higher the level the more defined things get — the more ‘differentiation’ occurs. The more integrated parts something contains the more defined something is. A single letter has arguably no parts and is (therefore) extremely flexible (undifferentiated) and can be part of a word which can mean literally anything. A word can be part of a sentence which can mean almost anything. An entire sentence can be part of a story which can mean many things. An entire paragraph is less flexible … and so forth. The idea is clear, I hope, that the higher the level, the more integrated parts, things get more and more defined, until, ultimately, the final form is stated. When we look at the Apollo 13 manual as a whole we intuitively understand that this has no bottom-up explanation. The existence of the manual cannot be causally explained from the level of the intrinsic (local?) functionality of letters. This is what Douglas Axe calls the universal design intuition. Summarizing: wholes exist well over and beyond the level of fermions and bosons, these wholes have a hierarchal structure and their internal levels have accumulative definition. And, importantly, all of this stuff cannot be explained bottom-up from the level of the parts. Returning to my point about the nature of ‘function’: in order to understand functionality one has to start at the level of the whole and work one’s way down. The function of a part can only be defined relative to the whole. Put another way: the function of a part is its role in the whole. A function is not an intrinsic property of a part, but it is an extrinsic property derived from the whole. Origenes

gpuccio @41:

The fact remains that the 2000+ superfamilies in SCOP classification are the best undeniable argument for design in biology.

That seems like the most verifiable conclusion one could draw from reading that thread, despite the focus being on something else. Thank you for commenting on it. Apparently you're the most qualified person to comment on that important subject in this site. At least that's my perception, which could be inaccurate. As far as I'm concerned, I've used the clear explanations you've written on the proteins subject in order to learn more about it, hence was missing your comments in that thread. However, I understand your valid reasons for not getting into that discussion. Mile grazie! Dionisio

EricMH: Thank you for the explanation. I agree with what you say, even if it is not completely clear to me how you translate the concepts of entropy ans enthalpy and free energy to the biological context. However, I think we agree on the final scenario, and that's the important thing. gpuccio

Dionisio: The SUPERFAMILY database is interesting, and I am experimenting with it. Regarding that thread, the focus seemed to be on common ancestry, and frankly I am tired to discuss that. The fact remains that the 2000+ superfamilies in SCOP classification are the best undeniable argument for design in biology. gpuccio

Eric Anderson: I agree with you. I agree with you on practically everything. I think we understand each other very well. :) A couple of points deserve, maybe, some further discussion. About Shannon entropy. I agree that we need a basic setup to make any discussion about that: a system were we can transmit bits, and some basic protocol to transmit them, receive them, compress them. I am not an expert of Shannon's theory, not at all, but I think that these basci "agreements" should be considered as part of the system we are analyzing. Then, inside that system, we can reason about transmission and compressibility of the intended message. What I am trying to say is that IMO you underestimate the objectivity of Shannon's entropy: as far as I can understand, it is an objective mathematical property of the message we have to transmit, even if it requires some basic agreements in the system. So, if a string is scarcely compressible, it is so because of its intrinsic properties, and not because of the way we have set the system. IOWs, a non compressible string is not compressible, however we set the system. On the contrary, a highly ordered string is always compressible, whatever the system You offer a counter-argument when you say: "There is no reason I can’t write an algorithm that contains a function that says “if I come across aljgoqjektnanksautyapley, then transmit the symbol &.” Ta da! Under the new “Anderson Method” of compressibility, the string aljgoqjektnanksautyapley turns out to be extremely and highly compressible — it can be transmitted with a single character!" I think that here you are making an error that deserves to be discussed. So, let's say that we have our basic settings for transmission and compression. Those settings are part of the system, so they can work for any string I want to transmit. Under those basic conditions, let's say that the string you propose: aljgoqjektnanksautyapley is random and completely not compressible. Therefore, if you want to transmit it, the only way is to give all the bits that describe it. Now, if you want, instead, to transmit an algorithm that can generate the string, that is an acceptable way to transmit the string itself. The algorithm you propose is to transmit: “if I come across aljgoqjektnanksautyapley, then transmit the symbol &.” And then transmit the single character &. That's OK, but obviously it is longer than simply transmitting the bits of the string itself, because those bits have however to be included in the algorithm. IOWs, the complexity of the "compressing algorithm" is higher than the complexity of the object we want to compress. So, the Kolmogorov complexity is the complexity of the string itself, because it is simpler than the complexity of the algoritthm. Now, let's make another example: the digits of pi. Let's say that the message we want to transmit is the first 100 digits of pi. Of course, here we are not interested in communicating what they are or what they mean as a mathematical constant. We just want to transmit the sequence. If we assume that such a sequence, even if certainly not random, is scarcely compressible (which I believe should be true), Its Shannon entropy will be very similar to the number of bits necessary to describe it. In alternative, we can simply transmit an algorithm which, according to the basic rules of the system, will generate the string as soon as received (just think in terms of a general Turing machine). Well, that algorithm will probably be longer than the string itself. So, we are apparently in the same situation as before: no real compression. But what is we want to transmit the first 10^15 digits of pi? In that case, an algorithm that can compute them is probably much shorter than the string itself. So, in that case, the Kolmogorov complexity of what we want to transmit would definitely be the complexity of the algorithm, and not the complexity of the string itself. As you can see, these are objective and very interesting properties of strings as such, and they do not depend on the basic settings of the system, even if they do require a basic setting of the system. I will try to summarize the above concepts: a) Some strings are scarcely compressible because of their intrinsic nature. In this group we find most purely random strings, but also many functional strings where the function does not require some special order in the string, but only the correspondence of the string to a higher level of understanding (for example, how a proteins folds and works at biochemical level, or how an ordering algorithm works). In general, I think that functional strings are more ordered than a purely random string (that's why we can sometimes recognize a language even if we don't know what it means). But in some cases that "intrinsic order" can be very low, and I think that protein strings are usually of that kind. b) Some strings are highly compressible because they are highly ordered. A string of 10^15 "1". or of 10^15 "01" is a good example. c) Other strings are compressible because, even if they are not ordered at all, they are computable. The sequence of digits of pi is a good example. d) For computable strings, here is an interesting aspect: the Kolmogorov complexity (which is what we are really interested in computing functional information) grows as the string becomes longer, but as soon as the complexity of the string equals the complexity of the algorithm that can compute it, the Kolmogorov complexity remains the same, while the length of the string continues to increase. That is very interesting, because the generating algorithm can be viewed as the specification of the string (after all, it can compute the string because it has been built using the information about what the string really is, in the case of pi the ratio of a circumference to its diameter). On the contrary, the length of the string is related to its potential complexity. So, if we consider simply the potential complexity, we have the strange result that the complexity can increase at will in a system that includes the generating algorithm. If we equal that potential complexity to the functional complexity, we would have the paradox of generating new functional complexity without any further conscious intervention in the system. But luckily that is not the case. Functional complexity corresponds to the functional Kolmogorov complexity of an object, not to the functional potential complexity. In the case of pi, the complexity apparently increases as the computed result becomes longer, but the specification (the generating algorithm) remains the same. So, the functional complexity in the system (the functional Kolmogorov complexity) remains the same, even if a constantly growing string corresponding to the specification (the ratio of a circumference to its diameter) is being generated in the system. OK, I apologize for this long and probably too complex discussion. I hope you will be patient enough to consider it. gpuccio

Eric Anderson and Origenes: Wow! This has really become an interesting discussion! :) Many thanks to you both for your precious contributions. Eric, I will comment on your thoughts in next post. For the moment, I would like to say that you have stolen my thoughts with this statement: "That said, I understand where you are coming from Origenes. What you are describing as function sounds like something we might call “purpose”." Well, you may believe it or not, but that is exactly what I was going to write to Origenes in my last post to him. In the end, I refrained, probably for fear of being misunderstood, or of opening a much wider discussion. But, as you said it yourself, I can happily agree. I absolutely agree also with all the further comments you offer in post #32. That's exactly what I think. I will only add that purpose is probably more than function, but that any wider purpose has to be implemented by specific functions. All discussions about purpose are extremely important and necessary. The only problem I see is that, like many basic philosophical problems, it could lead to some form of infinite (or at least very long) regress, where we end up asking the purpose of life, or of the universe itself. Nothing bad in that, after all. :) gpuccio

Eric Anderson @32:

I believe gpuccio has been focusing his research on identifying complex specified information at a more basic level than ultimate purpose. [...] it has the advantage of being more mathematically tractable and of setting a lower limit to what must be required for any higher level of functional organization.

Yes, agree. Perhaps this relates to the comment @35? Dionisio

gpuccio: Off topic: I expected to read your insightful comments in this thread: https://uncommondesc.wpengine.com/informatics/common-ancestry-bioinformaticist-julian-gough-on-the-superfamily-database-on-proteins-in-genomes/ Did I have wrong expectations? :) Dionisio

The term 'system' used in biology: https://en.wikipedia.org/wiki/Biological_system The same term in other widely known contexts: https://en.wikipedia.org/wiki/Operating_system https://en.wikipedia.org/wiki/Economic_system https://en.wikipedia.org/wiki/Political_system http://www.thefreedictionary.com/legal+system Dionisio

gpuccio @24:

[...] we can focus our attention, for practical reasons, on the “local” function of the protein, without considering, for the moment, the links to other levels of function in the cell.

That seems a very practical approach that is conducive to 'bit' quantification of structural novelties at the individual protein level. Higher functionality levels may be more difficult to quantify. Dionisio

@GPuccio, in biology, as entropy increases the chance of DNA being functional decreases, and CSI decreases. High entropy only means high CSI if it manages to hit a small, specified target. Alternatively, if increase in entropy invariably meant an increase in functionality, then this means the functional search landscape is extremely dense. An upshot of this is that genetic engineering would be quite easy. We can create random DNA sequences much more rapidly than nature, and should be able to easily hit the functional targets in the lab. However, this would still not equate to high CSI, since specificity is extremely large. The only way that arbitrary entropy increase can equate to functionality increase in a sparse functional landscape is if there are pathways to the functionality. But this requires large amounts of prior CSI in the construction of these pathways, and the entropy increase is not a contributing factor to the CSI. So, in whatever way it is correct to say entropy increase also increases functionality, entropy increase cannot equate to CSI increase. The two concepts are antithetical to each other by definition. CSI itself is not an alien concept invented by the ID movement. The concept shows up in many different disciplines, such as free energy in physics as I pointed out. But, it always introduces the same problem of being inexplicable in terms of chance and necessity, and leads to nebulous work arounds such as emergence and self organization. This is merely a matter of math, because by definition chance and necessity cannot produce CSI. All they can do is destroy CSI, and the 2nd law of thermodynamics with free energy is one example of this. EricMH

gpuccio: One other quick point: Let's take a random string: aljgoqjektnanksautyapley There is no reason I can't write an algorithm that contains a function that says "if I come across aljgoqjektnanksautyapley, then transmit the symbol &." Ta da! Under the new "Anderson Method" of compressibility, the string aljgoqjektnanksautyapley turns out to be extremely and highly compressible -- it can be transmitted with a single character! :) And if that were the extent of my algorithm, then when it came across the string "to be or not to be" it would have no way to compress it and would have to transmit the entire string. There is nothing magical about compression or compression algorithms. An algorithm is just built upon a statistical expectation of what it expects to see, given various parameters about anticipated types of messages, anticipated language, grammar, etc. In practice what makes a meaningful phrase more compressible than a random string is the understanding and expectations and skill of the algorithm writer in creating a good algorithm for the task at hand. We cannot simply "take for granted" that all the work and effort and understanding and information required for compression somehow exists in and of itself and that a given string contains more or less information based on its compressibility due to the intelligence and hard work of the programmer. ----- Based on our understanding of how language and sensical information arises and is encoded, we can draw a general observation that, in a given symbolic system, complex specified information generally lies between the extremes of order and randomness. But beyond very specific, well-defined examples we can't draw any general conclusions about having more or less information as we move away from a sense string and toward one or the other end of the spectrum. If my sense string were modified to be a string of random characters it would have no more information than if it were to decay to a string of repeating 'a's. The information content cannot be ascertained by compressibility. (I know you know this. I'm just brainstorming out loud here . . .) Eric Anderson

gpuccio, Origenes -- good discussion. If I may jump in, I think gpuccio is using both terms "function" and "system" in the standard scientific understanding of the words. That said, I understand where you are coming from Origenes. What you are describing as function sounds like something we might call "purpose". I'm reminded of Aristotle's four causes: the material cause, the formal cause, the moving cause, and the final cause (or purpose). We can certainly analyze biology across more than one of these. I believe gpuccio has been focusing his research on identifying complex specified information at a more basic level than ultimate purpose. This is a bit less accessible, perhaps, than a more macro view, but it has the advantage of being more mathematically tractable and of setting a lower limit to what must be required for any higher level of functional organization. Eric Anderson

gpuccio @19:

You example about pi is not appropriate, IMO. “write the first 100 digits of pi” is not a way of conveying the first 100 digits of pi. IOWs, it is not a “compression” of the first 100 digits of pi.

I apologize if my example wasn’t clear. I agree with you that for purposes of transmission we are interested in a compression algorithm. Let me press, however, on the example a bit, because you are selling it short too early and the example remains instructive, even insightful, if we think through it all the way.

But just giving the instruction: “write the first 100 digits of pi” is not good, because the instruction does not have the true information to do what it requires.

I agree that the transmitted instruction does not have within itself the information to do what is required. But that is exactly my point. When we are dealing with a compression algorithm, the transmitted message never has the information to do what is required. It is always the case that we have some kind of protocol or code or pre-agreed-upon algorithm to unpack the transmission and recreate the original string. I can most certainly construct a compression algorithm that would recognize the first 100 digits of pi and then if it comes across that, transmit the instruction “write the first 100 digits of pi”. This is no different than transmitting a mathematical instruction to compute pi, which itself has to be understood and run at the other end.

On the other hand, the instruction “write 01 50 times” is a true compression (provided that the simple instructions are understood as part of the basic transmission code).

Exactly. Having the instructions understood on the receiving end is always the issue. It is no different than my example of pi. I can compress and uncompress a string any way I want, as long as the instruction on both ends is understood. But there is no free lunch and whenever I transmit a compressed signal I have to also transmit the protocol or code or unpacking instruction (either contemporaneously or beforehand). The upshot of this fact which most people don’t recognize is this: For any reasonable-length message it typically takes more bandwidth to implement a code, compress the string, and transmit both of them than it would require to just transmit the uncompressed string in the first place. There is no inherent reduction in the quantity of what needs to be transmitted, just because I can think of ways to compress the string. Instead, the savings comes in when we have a very long message (e.g., a book versus a sentence) or when we do lots of transmissions over time. -----

Shannon’s theory is exactly about “reduction of uncertainty” and according to Shannon’s theory the values of entropy are different for different messages. That does not depend on external factors, but on intrinsic properties of the message sequence.

I think I understand where you are headed, but let me make sure. In order to implement any algorithm or to calculate the Shannon measure we must, as you mentioned at the beginning, have the code in place beforehand. In addition, we have to understand various parameters, such as the language we are dealing with, the likelihood of occurrence of particular symbols, etc. All of these things are external factors that impact how the algorithm operates and what kind of result we get, are they not?

However, that has something to do with order (ordered systems being more compressible, and easier to describe) . . .

I don’t dispute that a more ordered system – in the sense of more repetitive – would be more compressible. But, again, we have to ask ourselves how and why? It is more compressible because (1) we as intelligent beings are capable of recognizing the repetition, (2) we have layered a new code on top of the existing string, replacing repetitive elements with shorter elements and instructions to recreate the repetitive elements, (3) we have transmitted the code and the unpacking protocols to the recipient. Once all of those are in place then, yes, we can compress a repetitive string more than a less repetitive string. This is all well and good for what Shannon was interested in: namely faithful transmission and compression of the transmission. But when people like Dr. Swamidass start talking about information and hinting that we have more information because a string is less repetitive or less compressible, then something is amiss with their understanding.

. . . in discussions about Shannon entropy, that “code” is given for granted, and is not part of the discussion.

That it is “taken for granted” is precisely part of the problem when we talk about strings and, in particular, when we talk about Shannon. In the first place, when talking about Shannon we aren’t really dealing with information. But there is a second, more nuanced, issue I am trying to highlight in my last couple of comments to this thread. Namely, that there is a great deal of information being smuggled in behind the scenes. It is this additional information, operating behind the scenes, that gives us the impression that we have somehow compressed things and ended up with less information. No. That is not how compression works. What we have in fact done is add an additional layer of information – highly complex, specified information – on top of our string. And a more random string does not mean that it has inherently more information somehow. It is just less compressible or requires a better algorithm. -----

As I have tried to argue, function is often independent from intrinsic order of the system. The sequence of AAs for a functional protein may be almost indistinguishable from a random AA sequence, until we build the protein and see what it can do.

Agreed. Order, in terms of repetitiveness, is not equivalent to function. I fully agree there. To go the next step, I would add that complex specified information resides between the extremes of order and randomness, existing sometimes on a knife’s edge balancing act between contingency and order.

Now, I think that we should use “complexity”, in a generic and basic meaning, just as a measure of “the quantity of bits that are necessary to do something”.

I definitely agree that we can use the word in a generic and basic meaning. In fact, I would prefer a more basic definition than layering on a bit calculation. Yes, there are different ways to measure complexity and doing a bit calculation per Shannon or a modified Shannon is probably a reasonable surrogate to help us measure the complexity of a system. But I think we can talk about complexity in a generic sense, before determining which system we are going to use to measure it.

“Shannon entropy provides an absolute limit on the best possible average length of lossless encoding or compression of an information source.” So, this kind of “complexity” has nothing to do with meaning or function.

Absolutely agree. As did Shannon. :)

On the contrary, the “complexity” related to functional information measures the bits that are necessary to implement a function. As I have said, this is a concept completely different from compressibility and transmission. As functional information is often scarcely compressible, the functional complexity of a system will often be similar to the total complexity of a random system.

I’d like to understand a bit better what you mean by this. If we are going to implement a function in three-dimensional space we can go about it a couple of different ways. 1. We can specify the location and trajectory of each particle in the system. 2. We can use start points, end points, vectors, axes, geometry and math to describe the system. A CAD file would be an example of utilizing this kind of approach. (I think you’ve use this example before.) The second approach would typically be more compressible than the first, would it not? Whether it is rational to do #1 for a non-functional system and #2 for a functional system is a separate question I’ve been addressing in my comments. But in general, #2 would often be significantly more compressible than #1, correct?

In this case, we are rather interested in finding how much of the potential complexity of the system is really necessary to implement its function. That’s what we do by computing the rate of the target space to the search space. But that has nothing to do with compression based on order.

I’ll have to think about this some more. You’ve probably already done so, but can you give an example of what you mean here?

I think that too much importance is often given to terms like “complexity” and “information”. The only meaning of “information” that is interesting for ID is specified information, and in particular functionally specified information. All other meanings of “information” are irrelevant.

I think it would be difficult to give too much importance to information, as that is at the basis of everything we are talking about, but I understand what you are saying about definitional battles, and we are on the same page substantively. As a practical matter, I prefer not to give away the word to the detractors. I don’t want people thinking there are bunch of different kinds of information out there, and intelligent design just deals with one particularly esoteric kind. Then we are forced to have a subsequent battle as to whether our “information” is more important or more relevant to biology than their “information.” It is too easy for people who don’t understand design or who are actively critical of design, like Swamidass or Liddle, to jump on something like Shannon “information” without understanding what they are talking about and claim that it is the real information and that those ID folks are trying to push some strange idea called complex specified information. No. When I talk about intelligent design I am talking about information in the basic, understandable, dictionary meaning of the word. I am dealing with real information. And anyone talking about Shannon’s measure or claiming that information increases with disorder isn’t talking about information. In fact, they don’t know what they are talking about. So I am not willing to give away the word “information” to anyone who comes up with a nonsensical and illogical use of the word. I will continue to try to educate people on what information is, one step at a time. Hopefully it will help someone to actually understand the issues. All that said, I realize this is something of a tactical decision. I certainly understand and respect your approach, which I have taken myself at times. At the end of the day you are certainly correct that we are interested in complex specified information. Hopefully people will be willing to make the effort to understand what that is. Eric Anderson

... or maybe not. :) :) Upright BiPed

Origenes: OK, it's only a question of words, but words have some relevance to the reasonings, here, as I will try to explain. My concept of function is much simpler. If you look at my OP about functional complexity: https://uncommondesc.wpengine.com/intelligent-design/functional-information-defined/ you will see that I define function in a very simple way: anything that you can do with the object you are observing. The only requirement for a function is an explicit definition of the function itself. So, I can define for a stone the function of paper-weight, or just the function of being a rudimentary weapon, and so on. There is no concept of "serving" something else, just the concept of what can be done with the object in its current configuration. This empiric definition of function is vital, IMO, for the whole ID theory, because it is the only one that allows an objective use of the concept, and allows specific measurements of the complexity linked to each defined function. If you have read some of my posts here, you can understand that such an explicit and objective definition is vital for the reasonings I make here. In a sense, any scientific discourse must be done in what you call "a fitting context". No concept of science has any meaning if there are not people who can understand it, for example. But objective science must consider observable facts that are in some way independent from who makes the observation. So, an enzyme does what it does in the same way either in the cell or in the lab. That is a fact. So, I call function what it does, because it is something that can be done with the object, and it is objectively so. There is more: the reason why an enzyme does what it does lies in its specific configuration, which in the end depends on the AA sequence. Now, even if you consider that an enzyme does what is does because it "serves" the cell in some way, still it is able to do it exactly for the same reason that it is able to do the same thing in the lab: its configuration. When the enzyme catalyzes a reaction, it has usually no active part in what happens after, with the results of its activity. Therefore, the information in its configuration is essentially necessary to catalyze the reaction, and, usually, for nothing else. Therefore, it is obvious that we must use the "local function", as I call it, to measure the complexity linked to the configuration. The specific bits of functional information present in the sequence are necessary to catalyze the reaction, and not for anything else. The "serving" to the cell is due to other configurations, like the presence in the right context of many molecules that, for example, work in a cascade. But each molecule does what it does. The total result is the sum of the different "activities" (as you call them), but the intelligent information in each molecule is finalized to the specific activity, while the intelligent information in the design of the whole system (including all the molecules) is what ensures the final result. So, my use of function to define what an object does, by itself, is not only a verbal choice: it is a necessity. It is that "local function" that is supported by the configuration, and therefore the AA sequence. That is the digital information that we want to measure, because it measures the improbability of having that single object without a designed process. Of course, if we want to consider the functional information of the whole system (for example a protein cascade) we have to sum the functional complexity (in bits) of each component, and also to consider the information necessary to have all the components at the right place, at the right time, and so on. Dembski has tried something similar, but the task is really difficult, and anyway it has to rely in part on the measurement of the functional information in each component. So, I am in no way underestimating the importance of considering whole systems, and the functional information necessary to have them as a whole. I am only saying that all starts at the level of each single component, and each single component has its local function and is configured for that local function. That function is as much a function as the global function of the whole system, or of the cell, or of the organism. Because, in the end, the only simple and objective way to define a function for an object is: what can we do with this object? However, I absolutely respect your terminology and your way of thinking. I just wanted to explain the operative reasons that make me stick to mine. gpuccio

GPuccio: Thank you for your response.

For local function I mean a well defined function that the object can implement by itself, even if it is not connected to a wider context.

Instead of the term ‘local function’ I would suggest ‘activity’ or ‘operation’. I prefer ‘function’ to refer to the state of being subservient to something larger. A heart is functional / subservient to the organism. A function in this sense can never be ‘local’; can never be isolated from a larger context.

I have given the example of an enzymatic function. Even is an enzyme can be part of a wider context, and indeed it always is, you can take the protein, study it in the lab, and still it can implement its enzymatic activity, even if it is isolated by its natural context.

First, you still need to provide a context for the enzyme to implement its enzymatic activity. Second, in this new unnatural context it obviously cannot have the same function — because the cell is absent and function is context dependent — but it can have the same activity. Maybe we can find agreement here. I think that saying that the enzyme has the same activity in the cell and in the lab resolves my concern. For me it is incoherent to say that the enzyme has the same function — or ‘local function’ — in the cell and in the lab. You are free, of course, to continue to use the term ‘local function’. However I find that term incoherent — those two words contradict each other — and I will translate ‘local function’ with ‘activity’.

Indeed, I don’t understand why you say that “X is not functional if you cannot point out something larger than X.”

X is functional for something else. X has to serve, be useful, for something larger in order to be rightly termed “functional”. That’s what functional means.

How can that be true? Isn’t an isolated enzyme functional, for its ability to accelerate a specific reaction?

No, I believe it isn’t. Its activity is only functional if it serves a larger coherent context.

Isn’t a single phrase functional, if it conveys a specific meaning?

A phrase can only convey a meaning if it is situated in a fitting context of e.g. the communication between two persons. IOWs you are assuming a fitting context here. A single phrase on a piece of paper in the unfitting context of e.g. the uninhibited planet Draugr obviously conveys no meaning and therefore has no function. Origenes

Hello Origenes, GP, A paper you might find interesting for background. The Nature of Hierarchical Controls in Living Matter The paper appears below the fold. Upright BiPed

Origenes: For local function I mean a well defined function that the object can implement by itself, even if it is not connected to a wider context. I have given the example of an enzymatic function. Even is an enzyme can be part of a wider context, and indeed it always is, you can take the protein, study it in the lab, and still it can implement its enzymatic activity, even if it is isolated by its natural context. The same is true for a phrase that has a definite meaning. If I say: "The molecule of water is made of atoms of hydrogen and oxygen". I am conveying a specific meaning. Of course, that can be only the beginning of a wider discourse about water, or about many other things, but still the "local" meaning of the phrase remains. So, I would define "local function" as any function that the object we are analyzing can implement by itself, without any reference to a wider context. Indeed, I don't understand why you say that "X is not functional if you cannot point out something larger than X." How can that be true? Isn't an isolated enzyme functional, for its ability to accelerate a specific reaction? Isn't a single phrase functional, if it conveys a specific meaning? Another example could be a simple ordering algorithm, which orders the inputs it receives. It can certainly be part of a much more complex software, and indeed it almost certainly is, but can you deny that the code of the ordering algorithm, even if isolated, can still order inputs? I would like to understand better why you apparently don't accept this idea. gpuccio

Gpuccio, Thank you for your response. IMHO the term ‘function’ is always connected to a higher level — a larger whole or a ‘context’. X is not functional if you cannot point out something larger than X. IOWs X can be functional only if X is a component of something larger than X. Function implies hierarchy. Function only exists if there is a larger whole to serve.

GPuccio: However, that said, it is also true that we can focus our attention, for practical reasons, on the “local” function of the protein, without considering, for the moment, the links to other levels of function in the cell.

Could one argue that “local function” refers to the adjacent higher level of function? As in, the local function of a letter is being part of a word (which represents a higher level function), and the local function of a word is being part of a sentence (which represents a yet higher level function) and so forth?

GPuccio: Finally, I generally use the word “system” in a generic sense, not connected to functionality, because that is the common use of the word in physics. See Wikipedia: ...

Yes I understand. Nevertheless I would have preferred them to use another term. For me 'system' is connected to 'design' and 'function', so this choice of terms is mildly confusing. Origenes

Origenes: "I would like to note that being integral to a larger functional system* is a prerequisite for functional information." A very important point. And yet, I would say yes and no, for the reasons I am going to explain. Let's take, for example, a protein which has an enzymatic function. Of course, the enzymatic function is linked to many cellular contexts, and so the concept of functional coherence is very important. Moreover, there are some basic requisites without which no function can be performed: the protein must be located in the correct cellular environment, in the correct biochemical setting, pH, and so on. So, in that sense, it is dependent on the cellular system, not only because its function is linked to higher order functions, but also because its function could not work in any environment (say, in interplanetary space). In a sense, I would say that the concept of functional coherence is very similar (but not identical) to the concept of irreducible complexity. It is a pillar of ID theory, and I would never underestimate its importance. However, that said, it is also true that we can focus our attention, for practical reasons, on the "local" function of the protein, without considering, for the moment, the links to other levels of function in the cell. For example, for an enzyme, the local function can be easily defined as the ability to accelerate a specific biochemical reaction, provided that the basic environment is there (pH, temperature, substrate, and so on). This is useful, because we can usually demonstrate that the local function of a protein is already functionally complex enough to infer design. And, while higher level functions and connections certainly add to the functional complexity of that protein, they are more difficult to evaluate quantitatively. Therefore, it is often useful to compute the functional complexity of the local function in itself, which is certainly a lower threshold of the approximation of the whole functional complexity of that protein, and is much easier to compute. IOWs, you have to have a certain amount of functional complexity to be able to accelerate a specific reaction, even if you do not consider why that reaction is necessary in the cell, if it is part of a cascade, and so on. Finally, I generally use the word "system" in a generic sense, not connected to functionality, because that is the common use of the word in physics. See Wikipedia: https://en.wikipedia.org/wiki/Physical_system "In physics, a physical system is a portion of the physical universe chosen for analysis." I find the concept useful, because we can define any kind of physical system, and then analyze if there is any evidence of complex function and design inside that system. gpuccio

GPuccio: Functional information, the kind of information we deal with in language, software and biological strings, is not specially “ordered”. Its specification is not order or compressibility. Its specification is function. What you can do with it.

I would like to note that being integral to a larger functional system* is a prerequisite for functional information. This is what separates random pixels from pixels of a photograph, random letters from coherently arranged letters of the Apollo 13 manual and ice crystals from DNA. Axe on ‘functional coherence’:

What enables inventions to perform so seamlessly is a property we’ll call functional coherence. It is nothing more than complete alignment of low-level functions in support of the top-level function. Figure 9.3 illustrates this schematically for a hypothetical invention built from two main components, both of which can be broken down into two subcomponents, each of which can in turn be broken down into elementary constituents. Horizontal brackets group parts on a given level that form something bigger one level up, with the upward arrows indicating these compositional relationships. Notice that every part functions on its own level in a way that supports the top-level function. This complete unity of function is what we mean by functional coherence. [Douglas Axe, ‘Undeniable’, Ch. 9]

(*) IMHO the term ‘system’ should always be linked with functionality. I find the term ‘random system’ unhelpful. Origenes

EricMH: I am not sure I understand. How would that apply to a functional protein vs a random AA sequence? gpuccio

gpuccio @19:

I think that too much importance is often given to terms like “complexity” [...]

Agree, that's why I use "complex complexity" just to call it somehow, though it's still inaccurate. :) Dionisio

If the enthalpy (number of microstates) of a system increases or the entropy (number of macrostates) decreases, then CSI increases. Enthalpy is the complexity part of CSI and entropy is the specification. If enthalpy is low and entropy is low, such as a crystal, then there is no CSI. If enthalpy is high and entropy is high, there is also no CSI. Free energy is enthalpy minus entropy. This means that free energy is CSI, and evidence of design. EricMH

Eric Anderson: Interesting thoughts, but still I think differently on some points. There is no doubt that "a code of some kind is always required to convey information". I agree. But I think that, in discussions about Shannon entropy, that "code" is given for granted, and is not part of the discussion. You example about pi is not appropriate, IMO. “write the first 100 digits of pi” is not a way of conveying the first 100 digits of pi. IOWs, it is not a "compression" of the first 100 digits of pi. A compression of the first 100 digits of pi (or of the first 1000, or 10000) would be to transmit the algorithm to compute them. If the algorithm is shorter than the actual result, then it is a compression, in the sense of Kolmogorov compexity. But just giving the instruction: "write the first 100 digits of pi" is not good, because the instruction does not have the true information to do what it requires. On the other hand, the instruction "write 01 50 times" is a true compression (provided that the simple instructions are understood as part of the basic transmission code). Shannon's theory is exactly about "reduction of uncertainty" and according to Shannon's theory the values of entropy are different for different messages. That does not depend on external factors, but on intrinsic properties of the message sequence. However, that has something to do with order (ordered systems being more compressible, and easier to describe), but it has in itself no relationship with functional specification. As I have tried to argue, function is often independent from intrinsic order of the system. The sequence of AAs for a functional protein may be almost indistinguishable from a random AA sequence, until we build the protein and see what it can do. Now, I think that we should use "complexity", in a generic and basic meaning, just as a measure of "the quantity of bits that are necessary to do something". So, in Shannon's theory, complexity can be used to designate Shannon's entropy, which is a measure related to compressibility and transmission of a message. I quote from Wikipedia: "Shannon entropy provides an absolute limit on the best possible average length of lossless encoding or compression of an information source." So, this kind of "complexity" has nothing to do with meaning or function. On the contrary, the "complexity" related to functional information measures the bits that are necessary to implement a function. As I have said, this is a concept completely different from compressibility and transmission. As functional information is often scarcely compressible, the functional complexity of a system will often be similar to the total complexity of a random system. In this case, we are rather interested in finding how much of the potential complexity of the system is really necessary to implement its function. That's what we do by computing the rate of the target space to the search space. But that has nothing to do with compression based on order. I think that too much importance is often given to terms like "complexity" and "information". The only meaning of "information" that is interesting for ID is specified information, and in particular functionally specified information. All other meanings of "information" are irrelevant. And "complexity" just means, for me: "how many bits are necessary to do this?" "This" can be different things. In Shannon's theory, it is conveying a message. In ID, it is implementing a function. "Complexity" is, in all cases, just a qunatitative measure of the necessary bits. gpuccio

Just as a final side note on the above, I should add that there is a general principle at work here regarding the relationship between codes, communication and transmission. Many people are confused about this and some, even in the design community, have spoken too loosely about this. This confusion sometimes arises if we talk about sensical strings as though they existed in a vacuum. The principle is this: When we use a coded message to convey a particular piece of information, it ultimately requires that more information be conveyed (or already have been conveyed) to the recipient, not less. The use of codes is related to speed, efficiency over time, and ease of communication, not the overall quantity of information that must be conveyed to make sense of a single communication. The value of codes* (whether language grammar or otherwise), is that it allows us to set up a system beforehand that contains a great deal of underlying information. Then, in the moment of actual transmission, we can use our code to piggyback on that background information, vastly increasing our speed of communication. Also, as we repeatedly use our coded system, we gain tremendous efficiency over time. I hope that makes sense, but if not let me know and I'd be happy to elaborate further. ----- * It seems a code of some kind is always required to convey information, certainly at a distance without personal interaction. This deserves its own discussion another time. What I'm focusing on here is the implication for a single message or single string. Eric Anderson

gpuccio @14: Thanks for the good thoughts. We still need to be careful, though, about the idea that entropy drives toward complexity (which, as you note, is sometimes poorly defined).

You are right about the “uniformity” of entropic disorder. But the point here is again that a word is ambiguous, and this time the ambiguous word is “complexity”. A random system, even if uniform for all practical purposes, still can be said to have a lot of “unspecified complexity”, because if you want to describe it exactly you have to give information about each random and contingent part of it. In that sense, gas particles are not so different from random strings. However, that complexity is completely uninteresting, because for all practical purposes random gas states and random string behave in the same way: that’s the “uniformity”.

What would make a uniform sample more complex than a non-uniform sample? Why would I have to describe each gas particle in a uniform sample, but not in a non-uniform sample? Take my container with water example. One could argue that if we wanted to fully and completely describe the uniform sample, we would -- as you say -- have to describe each particle, its composition and position within the sample. But this is also true of the non-uniform sample. *And* in the uniform sample, we have one less differentiating factor -- namely the heat content (or movement speed of the molecules). Thus, we have one less factor to describe. Similarly, if we do a more high-level description at the macro level, the same holds true: it is easier and shorter to describe the uniform sample than the non-uniform sample. In either case, the entropic drive toward uniformity leads to less complexity, not more. ----- Again, I think in the case of strings this is harder to see, because information entropy doesn't drive toward uniformity in the sense of an actual physical reality. An "a" doesn't decay to a "b" and so on. Once something changes our "a" to a "b" it stays a "b" until the something changes it again. (BTW, we might do well to ask what the "something" is that is changing our string?) Information entropy drives toward uniformity in the sense of nonsense or randomness. So one could easily argue that a nonsense string can be described, for practical purposes and intents and real-world applications, as "a string of random characters of length x". That is a very simple way to describe it. ----- Think of it this way: One issue is whether we believe we have to transmit or reproduce the precise string in question. In that case, one could argue that a purely random string is more complex because it takes more instructions to transmit or reproduce. This is true enough. This, unfortunately, is where most people stop in their analysis. However, and this is important: 1. The primary reason why we can describe a sensical string with a shorter description than a purely random nonsensical string is because the necessary background information has already been conveyed to the recipient. For example, if I have a string of digits that represents the first 100 digits of pi, I can just convey the message "write the first 100 digits of pi", which is much shorter than conveying the entire string (which I would have to do with a pure random string). But note well that my shorter conveyance is built upon, contingent upon, and only works because of the fact that the recipient already (i) understands English and the rules of grammar related to my conveyance, (ii) understands what pi is, (iii) understands how to compute the first 100 digits or where to look up the information elsewhere, and (iv) knows how recreate the string. So while I think I have conveyed a tremendous amount of information with my short description, what I have really done is convey a small amount of information, which piggybacks on a whole background suite of information that was already previously conveyed to the recipient (by me or someone else). There is no free lunch. Substantively, one way or another the recipient has to receive these 100 digits of pi. It seems short and simple in the moment of transmission, but only because almost all of the necessary information was previously conveyed and already in place. Indeed, it is clear when we analyze what is really going on that, over time, much more information was required to be transmitted to the recipient in order to be able to deal with our short, cryptic transmission, than if we had just transmitted the actual string. 2. Shannon was concerned about transmission and channel capacity, not information. Regardless of the unfortunate term and confusion surrounding "Shannon information," Shannon himself said he wasn't dealing with information per se. I know we're on the same page here, but I mention this again because when we are talking about entropy and descriptions of strings and what might be required to convey a particular string, it is very important to keep in mind that we are not talking about information. Rather, we are talking about channel capacity and what is required for faithful transmission in a particular instance, given (a) a desire for precise and accurate replication of the string on the receiving end, and (b) the various protocols, grammar, and background knowledge of the recipient that are already in place. Eric Anderson

KF: Thank you for the very pertinent clarification! :) gpuccio

Folks, even so humble a source as Wiki has a useful insight, in its article on Informational Entropy, as I have cited several times:

At an everyday practical level the links between information entropy and thermodynamic entropy are not close. Physicists and chemists are apt to be more interested in changes in entropy as a system spontaneously evolves away from its initial conditions, in accordance with the second law of thermodynamics, rather than an unchanging probability distribution. And, as the numerical smallness of Boltzmann's constant kB indicates, the changes in S / kB for even minute amounts of substances in chemical and physical processes represent amounts of entropy which are so large as to be right off the scale compared to anything seen in data compression or signal processing. But, at a multidisciplinary level, connections can be made between thermodynamic and informational entropy, although it took many years in the development of the theories of statistical mechanics and information theory to make the relationship fully apparent. In fact, in the view of Jaynes (1957), thermodynamics should be seen as an application of Shannon's information theory: the thermodynamic entropy is interpreted as being an estimate of the amount of further Shannon information needed to define the detailed microscopic state of the system, that remains uncommunicated by a description solely in terms of the macroscopic variables of classical thermodynamics. For example, adding heat to a system increases its thermodynamic entropy because it increases the number of possible microscopic states that it could be in, thus making any complete state description longer. (See article: maximum entropy thermodynamics.[Also,another article remarks: >>in the words of G. N. Lewis writing about chemical entropy in 1930, "Gain in entropy always means loss of information, and nothing more" . . . in the discrete case using base two logarithms, the reduced Gibbs entropy is equal to the minimum number of yes/no questions that need to be answered in order to fully specify the microstate, given that we know the macrostate.>>]) Maxwell's demon can (hypothetically) reduce the thermodynamic entropy of a system by using information about the states of individual molecules; but, as Landauer (from 1961) and co-workers have shown, to function the demon himself must increase thermodynamic entropy in the process, by at least the amount of Shannon information he proposes to first acquire and store; and so the total entropy does not decrease (which resolves the paradox).

KF kairosfocus

Eric Anderson: Thank you for your very good thoughts. I completely agree with you. You raise two important points. and I will comment on both: 1) You are right about the "uniformity" of entropic disorder. But the point here is again that a word is ambiguous, and this time the ambiguous word is "complexity". A random system, even if uniform for all practical purposes, still can be said to have a lot of "unspecified complexity", because if you want to describe it exactly you have to give information about each random and contingent part of it. In that sense, gas particles are not so different from random strings. However, that complexity is completely uninteresting, because for all practical purposes random gas states and random string behave in the same way: that's the "uniformity". Ordered states are often more "compressible", and they can be described with shorter "information". So, Swamidass here is equivocating on the meaning of both "information" and "complexity". If you use those two concepts without any reference to some sort of specification, it is easy to say that random systems or random strings exhibit a lot of (basic, unspecified, useless) information and complexity. But who cares? All random strings are similar because they convey no useful meaning, just as all random states of a gas are similar because they behave in a similar way for all practical purposes. That's what characterizes all random states of any complex system: a) they behave in a similar way, and they convey no useful information linked to their specific configuration. b) They are common, extremely common, super-extremely common. While ordered or functional states are exceedingly rare. In a few words, that is really the main idea in the second law, either applies to physics or more generally to all informational systems. "non interesting" states are so much more common then any other, that systems governed only by natural laws will ultimately tend to them. 2) Here you raise a very important point: order is different from function. Sometimes the two may coincide, but usually they don't. Functional information, the kind of information we deal with in language, software and biological strings, is not specially "ordered". Its specification is not order or compressibility. Its specification is function. What you can do with it. That is a much stronger concept than simple order. While you can have order from physical laws, you cannot have complex functional information from physical laws. The sequence of a protein is fully functional, but it is usually scarcely compressible. That's why most complex functional information is, apparently, "pseudo-random": you cannot easily derive the function simply from mathematical properties of the sequence itself, like compressibility. Indeed, the function is linked to some higher level knowledge: a) For language, that knowledge is understanding of the symbolic meaning of words. b) For software, that knowledge is understanding of how algorithms operate. c) For a protein sequence, that knowledge is understanding of biochemical laws, of how a protein folds, and of what it can do once it folds. None of that "knowledge" can be derived simply from properties of the sequence. All those forms of knowledge imply higher level understanding, the kind of understanding that only a conscious, intelligent and purposeful being can have. That's all the "magic" of ID theory: it allows us, by means of the concept of complex functional information, to safely detect the present of higher knowledge and purpose in the observed physical configuration of a system, a higher knowledge whose only possible origin is a conscious, intelligent, purposeful being. gpuccio

gpuccio @4: Good thoughts and comments. I have no doubt that he is confused about "Shannon information" and that this is a large part of his mistake. This is an issue that has caused no small amount of confusion and has tripped up many a would-be traveler in these waters. May whoever first used the term "Shannon information" promptly apologize from the grave . . . I agree with you that this is a big part of the issue, as you well highlighted. Let me offer two clarifications, however, so that everyone is on the same page: ----- Entropy (in the sense of the Second Law) does not necessarily mean that we end up with a system that is more complex. Indeed, entropy typically (and ultimately) drives toward uniformity. Entropy is often said to drive toward "disorder," but that is in the sense that it drives against functional organization, as well as against the preservation of gradients. Consider a container filled with a barrier in the middle, filled with hot water on one side and cold water on the other. When we remove the barrier, the gradient will quickly disappear and we will soon be left with a more uniform system, not a more complex one. (The same thing would happen with the barrier left in place, just much more slowly.) This issue is a little harder to see when we talk about strings of letters because they are not subject to the physical realities controlled by the Second Law. (The broad principles behind the Second Law apply to information, to functional constructions, etc. and we can properly talk about "entropy" in those contexts, but it sounds like Dr. Swamidass is talking about the Second Law in a more classical sense.) ----- One other point, when considering information, in the sense of strings of characters/symbols: Complex specified information typically lies between the extremes of uniformity and randomness. Sometimes we hear (and I have occasionally heard ID proponents incorrectly suggest) that more order = more information, or that less order = more information. Either may be true in a particular well-defined case, but neither is true as a general principle. Rather: - On one end of the spectrum we have uniformity. This is the law/necessity side of the spectrum. - On the other end we have randomness. This is the chance side of the spectrum. - In the middle is where complex specified information resides. Neither the result of simple law-like necessity, nor the result of random chance. Rather, a careful balancing act of contingent organization and complexity. Thus, just as design in the physical, three-dimensional world can be seen as a third real cause, juxtaposed against chance and necessity, so too design in the form of complex specified information can be seen juxtaposed against chance and necessity in a string. Eric Anderson

Understanding has nothing to do with information. - J. Swamidass Mung

Swamidass turns out to be a fool. An expert fool. Mung

Dr Swamidass, there are four people here now that are quite capable of conversing with you on this topic. If you do not believe you are equivocating on the issue, or do not understand why anyone would say that you are, then please jump in. More directly, you are misleading your readers. You are doing so as a scientist, and as a theist. We can quite easily set the theism aside, in that regard you are free to do as you wish. However, you are completely and quite carelessly wrong about the science. You should address it. Upright BiPed

Hello Eric, I followed the links and read Dr. Swamidass's paper. He does exactly as you say. Its quite a spectacle that he assumes the role of an authoritative scientific voice, and yet he completely misunderstands the material. Perhaps he'll stop by and defend his position. Upright BiPed

GPuccio: It’s the old misunderstanding, all over again.

Why am I not surprised? Swamidass, who claims to have been an “intelligent design fanatic”, is nevertheless plagued by his misunderstandings about ID. It’s hard to keep up, but here are some of them: Swamidass holds that ID “invokes God.” Swamidass holds that ID and SETI use a completely different methodology, and that he could name “five material differences” — but refuses to tell us what they are. Swamidass holds that cancer evolves and casts serious doubts on intelligent design. Swamidass holds that Behe’s 2004 paper contains “two clear errors” — but refuses to tell us what they are. Origenes

Hi GP,

It’s the old misunderstanding, all over again.

Of course it is...or is it? I do not believe there is any way that Dr. Swamidass isn't aware of the profound equivocation he is promoting to the public. If he is not serving the public as a responsible voice for science, then what is his motive for so clearly misleading his readers? Upright BiPed

To Whom This May Concern: Please, read carefully gpuccio's comment @4 for real clarification of this issue. Dionisio

Thanks, UB. I'm just now seeing this thread. If Professor Swamidass actually said this:

Did you know that information = entropy? This means the 2nd law of thermodynamics guarantees that information content will increase with time, unless we do something to stop it.

then he does not understand the issues. I second your request @2 for him to clarify what he means. Hopefully he will stop by. My hunch, just based on the one paragraph quoted from him is that he is conflating the general idea of increased "disorder" often discussed in the context of entropy with increased complexity. A common, but mistaken notion. Add to that, he is likely mistakenly thinking that objects contain information by their mere existence. Also a very common misconception. If he has those two misconceptions firmly in mind, we can see how he might think that entropy => complexity => information. This is completely wrong, but that's my guess as to what he might have been thinking. ----- More importantly, as noted elsewhere in these pages, even if we were to inappropriately grant his claim, whatever so-called "information" he is talking about has absolutely nothing to do with real, identifiable, transmittable, translatable, functional information -- the kind of information relevant in the real world and in biology in particular. Eric Anderson

UB: It's the old misunderstanding, all over again. Swamidass is using "information" in the Shannon sense. That has nothing to do with the concept of specified information or functional information, which is the only concept relevant to ID theory. It is obvious that a generic concept of "information", without any reference to a specification of any kind, can only mean the potential number of bits necessary to describe correctly something (or, as in Shannon, to convey that message). In that sense, a random sequence has more "information" than an ordered sequence. For example, if we have a random sequence of 100 bits, and we want to correctly describe it, we need all or almost all the bits to do that. But if we have a sequence made of 01 repeated 50 times, we can describe it just saying: write "01" 50 times. That is shorter than giving 100 bits. In that sense, random systems are more "complex" than ordered systems. In that sense, the second law says that the entropy in a system can only stay the same or increase, and therefore random configurations are destined to increase, according to natural laws. In that sense, order is constantly eroded by natural laws. But that has nothing to do with specified or functional information. Specified information is not random. That's why we call it "specified" or "functional": to clearly distinguish it from that kind of "information" which is not specified nor functional, that kind of "information" which is not information in our human sense at all, just a form of entropy. While random configurations are destined to increase by the work of natural laws, functional configuration are destined to decrease by natural laws. For the same exact reason. Order and function are constantly eroded by the second law. Especially function. Because, while some simple form of order can arise by natural laws at the expense of other forms of entropy, complex function never arises by natural laws. Complex function is always the product of conscious design. So, Swamidass and others can go on equivocating on the meaning of "information" (a world as ambiguous and abused as that other term, "love"). Those "reflections" have nothing to do with functional information, and with functional complexity. They have nothing to do with ID. They are meaningless word games. gpuccio

Upright BiPed, Perhaps Dr S. meant anything out there except complex functional specified information? Maybe something like the firmament? Anyway, in any case, didn't somebody say that nonsense remains nonsense regardless of who says it? Dionisio

Dr Swamidass, I understand you stop by this site from time to time. I have a quick question for you. Can you please pick an example and tell me any of this "information content" that will "increase with time, unless we do something to stop it"? Thanks. Upright BiPed

Eric Anderson, Did you see this?!?

“Did you know that information = entropy? This means the 2nd law of thermodynamics guarantees that information content will increase with time, unless we do something to stop it.”

Utterly and totally clueless. An esteemed member of the scientific community completely misunderstands the category and subject matter he's talking about. Upright BiPed