Complex Specified Information Design inference ID Foundations Information

Intelligent Design Basics – Information – Part III – Shannon

Spread the love

In this post I want to consider another aspect of information.  Specifically, I want to consider the concept of “Shannon information.”

First of all, I admit to having ruffled a few feathers when I mentioned in passing in a prior post that “Shannon information is not really information.”  As I have also written before in comments on UD, I don’t begrudge anyone referring to the Shannon metric as “information.”  That terminology has penetrated the English language and has become regularly-used in information theory.  So, no, I am not going to police everyone who puts the words “Shannon” and “information” next to each other.

However, no small amount of misunderstanding has resulted from the unfortunate term “Shannon information.”  In particular, as it relates to intelligent design, some critics have seized on the idea of Shannon information and have argued that because this or that computer program or this or that natural process can produce a complex string or a complex sequence, that therefore such a program or process is producing new complex “information.”  This proves, the argument goes, that purely natural processes can produce new and large amounts of information, contra the claims of intelligent design.

Such thinking demonstrates a lack of understanding of CSI – in particular the need for specification.  However, a large part of the problem results from the use of the word “information” in reference to the Shannon metric.  As I have stated before, somewhat provocatively, we would all have been better off if instead of “Shannon information” the concept were referred to as the “Shannon measurement” or the “Shannon metric.”

Claude Shannon published a paper entitled “A Mathematical Theory of Communication” in the July 1948 volume of The Bell System Technical Journal.  This paper is available online here and is considered a foundational groundwork for not only Shannon’s subsequent research on the topic, but for information theory generally.  To be sure, there are many other aspects of information theory and many other individuals worthy of acclaim in the field, but Shannon is perhaps justifiably referred to as the father of information theory.

But before delving into other details in subsequent posts, time permitting, I want to relate a short experience and then a parable.  Consider this a primer, a teaser, if you will.

The Warehouse

When I was a teenager in high school, one of my part time jobs was working in a warehouse that housed and sold equipment and materials for the construction industry.  On a regular weekly schedule we would load a truck with supplies at the main warehouse and drive the truck to a smaller warehouse in a different city to supply the needs in that locale.  The day of the week was fixed (if memory serves, it was generally a Friday) and the sending warehouse foreman made sure that there were enough people on hand in the morning to pull inventory and load the truck, while the receiving warehouse foreman in turn ensured that there were enough people on hand in the afternoon to unload the truck and stock the inventory.

Due to the inevitable uneven customer demand in the receiving city, the needs of the receiving warehouse would vary.  With good inventory management, a large portion of the receiving warehouse’s needs could be anticipated up front.  However, it was not uncommon for the receiving warehouse to have a special order at the last minute that would necessitate removing a large crate or some boxes from the truck that had already been loaded in order to make room for the special order.  At other times when no large orders had been made, we would finish loading all the supplies and find that we still had room on the truck.  In this latter case, the sending foreman would often decide to send some additional supplies – usually a high turnover item that he knew the receiving warehouse would likely need shortly anyway.

In either case, the goal was to make most efficient use of the time, money and expense of the truck and driver that were already slated to head to the other town – taking the best possible advantage of the previously-allocated sunk costs, if you will.  Ensuring that the shipment container (in this case a truck) made best use of the available capacity was a key to efficient operations.

I want to now take this experience and turn it into a parable that relates to Shannon information.

The Parable of the Fruit Truck

Let’s assume that instead of heating and cooling equipment and supplies, the warehouse sells fruit directly to customers.   Let’s further assume that the various kinds of fruit are shipped in different-sized boxes – the watermelons in one size of box, the pineapples in another, the apples in another, and the strawberries in yet another.

Now, for simplicity, let’s suppose that customers purchase the fruit on a long-term contract with a pre-set price, so the primary variable expense of the warehouse is the expense of operating the truck.  The warehouse would thus be highly incentivized to maximize the efficiency of the truck – sending it out on the road only as often as needed, and maximizing the carrying capacity of the truck.

The dock workers in our parable, however, are not particularly sharp.  As the fruit comes in from the farms, the dock workers, without confirming the contents, simply start packing the boxes at the front of the truck, working their way to the back.  Invariably, there are gaps and open spaces as the various-sized boxes do not precisely conform to the internal capacity of the truck.  Some days are better than others by dint of luck, but the owner quickly realizes that the packing of the truck is inefficient.  Worse still, customers regularly complain that (i) the truck is arriving only partly filled, (ii) boxes contain the wrong kind of fruit, or (iii) in particularly egregious cases, the boxes contain rotten fruit or no fruit at all.

As a result, the warehouse owner decides to hire a sharp young man fresh from the university whose sole job it is to figure out the best way to pack the truck, to create the most efficient and time-saving way to deliver as much fruit as possible given the carrying capacity of the truck.

Let’s say this young man’s name is, oh, I don’t know, perhaps “Shannon.”

Now our hero of the parable, Shannon, works in the office, not the loading dock, and is unable to confirm the actual contents of the boxes that are loaded on the truck.  Further, he quite reasonably assumes the dock workers should be doing that part of the job.  Notwithstanding those limitations, Shannon is a sharp fellow and quickly comes up with a formula that gives the owner a precise calculation of the truck’s carrying capacity and the exact number of each type of fruit box that can be loaded on the truck to ensure that every square inch of the truck is filled.

Elated with the prospect of putting all the customer complaints behind him, the warehouse owner hands down the instruction to the dock workers: henceforth the truck will be packed with so many watermelon boxes, so many pineapple boxes, so many apple boxes and so on.  Furthermore, they will be packed according to Shannon’s carefully worked out order and placement of the boxes.

After the next week’s shipments, the owner is surprised to receive a number of customer complaints.  Although not a single customer complains that the truck was only partly full (it was packed tightly to the brim in all cases), several customers still complain that (i) boxes contain the wrong kind of fruit, or (ii) in particularly egregious cases, the boxes contain rotten fruit or no fruit at all.

Furious, the owner marches to Shannon’s desk and threatens to fire him on the spot.  “I hired you to figure out the best way to pack the truck to create the most efficient approach to delivering as much fruit as possible!  But I am still swamped by customer complaints,” he fumes as he throws down the list of customer complaints on Shannon’s desk.  Unfazed, Shannon calmly looks at the customer complaints and says, “I understand you used to get complaints that the truck was only partially filled, but I notice that not a single customer has complained about that problem this week.  You hired me to find the most efficient delivery method, to ensure that the truck was maximizing its carrying capacity of boxes.  I did that.  And that is all I have ever claimed to be able to do.”

“But some of the customers got the wrong fruit or got no fruit at all,” sputters the owner.  Based on your work we told them they would be receiving a specific quantity of specific types of fruit each week.

“I’m sorry to hear that,” retorts Shannon, “but you should not have promised any specific fruit or any particular quantity of fruit based on my formula alone.  From my desk I have no way of knowing what is actually in the boxes.  The supplier farms and dock workers can answer for that.  What is in the boxes – what is actually delivered to the customer – has nothing to do with me.  I have no ability from where I am sitting, nor frankly any interest, in guaranteeing the contents of the boxes.  My only task, the only thing I have ever claimed to be able to do, is calculate the maximum carrying capacity of the truck with the given boxes.

The Analogy

The fruit truck is obviously but a simple and fun analogy.  However, it does, I believe, help newcomers get a feel for what Shannon can do (analyze maximum carrying capacity of a delivery channel) and what Shannon cannot do (analyze, confirm, understand or quantify the underlying substance).  We’ll get into more details later, but let’s kick it off with this analogy.

What similarities and differences are there between our parable of the fruit truck and Shannon information?  What other analogies are you familiar with or perhaps have yourself used to help bring these rather intangible concepts down to earth in a concrete way for people to understand?

 

118 Replies to “Intelligent Design Basics – Information – Part III – Shannon

  1. 1
    Eric Anderson says:

    Nota bene:

    I drafted this post more than two months ago, meaning to work on it more before publishing.  Unfortunately, work obligations and life generally have prevented me from spending further time on it.  As a result, I am publishing it “as is” in the hope it might nevertheless serve as a point for discussion.

  2. 2
    Neil Rickert says:

    As I have also written before in comments on UD, I don’t begrudge anyone referring to the Shannon metric as “information.”

    Just to be clear, the metric is not the information. The metric is a way of measuring the amount of information (or the channel capacity).

    Such thinking demonstrates a lack of understanding of CSI – in particular the need for specification.

    I am inclined to say that all Shannon information is specified. Noise signals are not usually considered to be Shannon information.

    I want to now take this experience and turn it into a parable that relates to Shannon information.

    Your parable is a poor illustration. Shannon was concerned with correct transmission of information over a noisy channel. Your parable ignores the concern with correctness. It is perhaps an analogy for a noisy channel, but it is misleading as an account of Shannon information.

  3. 3
    Upright BiPed says:

    Neil, who could know where to start with your comments in #2.

    Listen to what Claude Shannon is telling you:

    The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages. The system must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design.

    He is telling you that there is a message to be communicated from one point to another point, where it will be selected at that point. It will be selected — because it is specified, Neil. Representations have to be related to their meaning at the receiver in order for the message to be communicated. It’s the distinction between me writing a meaningless scribble on a piece of paper, or, writing the word “apple”. You recognize the “apple” because it’s specified; you know what those particular scribbles mean. Shannon is telling you that this process of selection at the receiver is irrelevant to the problem of engineering a communication channel that can carry any scribble – the one selected and all the others that are not, i.e. the noise. That is what the system has to be able to do in order to function.

    I am inclined to say that all Shannon information is specified. Noise signals are not usually considered to be Shannon information.

    You need a complete overhaul.

  4. 4
    Upright BiPed says:

    Ahh. Forgot the two words that clarify the point:

    He is telling you that there is a message to be communicated from one point to another point, where it will be selected at that point [as well]. It will be selected — because it is specified

  5. 5
    kairosfocus says:

    EA et al, yes, Shannon Info is really an info carrying capacity metric. In particular the entropy in a message is the average info capacity per symbol, which is linked to the info approach to entropy. (Entropy, on such a view, is a metric of average missing info to specify microstate given only a macro-level description of a thermodynamic system; which forces us to treat the microstate as effectively rndom, in trying to extract work, etc.) That is why functionally specific complex organisation and associated info is a needed further step. KF

  6. 6
    kairosfocus says:

    UB: Yes, the detection of a communicated message in the face of the interference of noise is a case of inference to design, to intelligent message rather than noise. We need to ponder the significance of the metric signal to noise power ratio that implies that there are characteristic . . . empirically characteristic . . . differences between overwhelmingly typical messages and overwhelmingly typical noise. This also appears in filter theory where noise as a rule is high frequency “grass” on a CRO screen — in the case of the classic Telequipment D52 (I think eventually bought out by Tektronix), that was very literal as the screen is bright green. Signals tend to be much narrower band, centred on a carrier wave. And yes I am familiar with frequency hopping and other spread spectrum techniques. They boil down to imposing a code that if known allows us to pull a signal almost like magic out of what would otherwise appear to be noise. Just think of the old electronics rule of thumb to avoid differentiators like the plague as this is a noise amplifying process. KF

  7. 7
    GBDixon says:

    The comments are correct: The set of valid messages among the total of all possible messages that can be received defines the information. Channel capacity has no meaning without this message list and information entropy and the channel capacity cannot be calculated without knowing the number of valid messages among the number of possible messages.

    When both a sender and receiver are involved, they decide together the set, or list, of valid messages: THEY decide what the information is. In SETI and molecular biology, there may be senders but their decisions regarding messages must be inferred as we cannot establish an a priori message list. In SETI we look for message sequences that are unlikely to be caused by natural events.

    In biology there are DNA sequences that are clearly information because they code for functional proteins. Some other sequences may not be clearly information, and we can argue whether they are or not, so the exact information content of a gene may be unknown but it has a lower bound. The set of all possible DNA sequences constitute the possible messages that can be received (by the molecular machines that interpret the sequences), and we can calculate an entropy bound as well.

    Shannon did more than determine channel capacity. He gave a fundamental definition of what constitutes information. This applies to all information. That is why his work is considered groundbreaking.

    In Eric’s fruit truck example there are more than one receivers and more than one set of information. To the woman who calculates truck gas consumption, the information is the weight of the load and the length of the route. To the person who loads the truck, the information is how much space is left. To the person who unloads the truck it is how much more is left to unload before I can take my coffee break. To the customer, it is what is wanted.

    To the gas purchaser Mr. Shannon has maximized the truck’s capacity, but by only delivering a small subset of what the customer wanted, Mr. Shannon’s delivery is NOT what the truck is capable of, and he fails in his mission to maximize channel capacity for information as defined by the customer.

    The truck is a special case and therefore kind of a poor example because it is capable of delivering, unchanged, a complete load of exactly what the customer wants without corruption. There is no ‘noise’ that corrupts the shipment(except maybe spoilage). We usually presume that the sender only sends the things the receiver (customer) wants and that something bad happens to the messages along the way. In biology, this is usually considered to be the DNA mutations that creep in and do nothing or cause problems.

  8. 8
    Eric Anderson says:

    Neil @2:

    The metric is a way of measuring the amount of information (or the channel capacity).

    Well, which is it, the information or the channel capacity?

    The only way you can say the metric is measuring “information” is to redefine information to mean the same thing as “channel capacity.” In which case we have then lost the value of the word “information” and we need to come up with a different word to describe what we originally used to think of as information.

    The whole term “shannon information” causes more confusion than light. That is precisely why an analogy is helpful for people to start grasping what we are dealing with.

    Again, as I said, I’m not going to change decades of people referring to the shannon metric as “shannon information.” That is too ingrained and there is too much inertia.

    But we need to be very clear when talking about shannon so-called information that we are not talking about the key aspects that we typically think of when we use the word information. We are not talking about semantics, vocabulary, meaning, intent, purpose, informing. All those, as Shannon himself clearly stated (and as UB pointed out above) are irrelevant to the shannon metric.

    I am inclined to say that all Shannon information is specified.

    Then you do not understand specification, as it is used in the intelligent design context. Specification involves meaning, purpose, function — precisely the kinds of things Shannon said were irrelevant to his channel capacity problem.

  9. 9
    bornagain77 says:

    The argument from information sure has come a long way on UD. Years ago on UD I remember many neo-Darwinists denied information was even in the cell. In fact in the past, this following comment has been used far more than once on UD,,,

    Information Theory, Evolution, and the Origin of Life – Hubert P. Yockey, 2005
    Excerpt: “Information, transcription, translation, code, redundancy, synonymous, messenger, editing, and proofreading are all appropriate terms in biology. They take their meaning from information theory (Shannon, 1948) and are not synonyms, metaphors, or analogies.”
    http://www.cambridge.org/catal.....038;ss=exc

    Then after a while, when some of the Darwinists’ realized that information actually was in the cell, and that it was not an ‘illusion’, their arguments ‘evolved’ to say that the information in the cell was merely Shannon information, and therefore, using Shannon’s broad metric for information, information, the Darwinists held, can therefore increase with Darwinian evolution,,,

    The Evolution-Lobby’s Useless Definition of Biological Information – Feb. 2010
    Excerpt: By wrongly implying that Shannon information is the only “sense used by information theorists,” the NCSE avoids answering more difficult questions like how the information in biological systems becomes functional, or in its own words, “useful.”,,,Since biology is based upon functional information, Darwin-skeptics are interested in the far more important question of, Does neo-Darwinism explain how new functional biological information arises?
    http://www.evolutionnews.org/2.....ss_de.html

    Three subsets of sequence complexity and their relevance to biopolymeric information – Abel, Trevors
    Excerpt: Three qualitative kinds of sequence complexity exist: random (RSC), ordered (OSC), and functional (FSC).,,, Shannon information theory measures the relative degrees of RSC and OSC. Shannon information theory cannot measure FSC. FSC is invariably associated with all forms of complex biofunction, including biochemical pathways, cycles, positive and negative feedback regulation, and homeostatic metabolism. The algorithmic programming of FSC, not merely its aperiodicity, accounts for biological organization. No empirical evidence exists of either RSC of OSC ever having produced a single instance of sophisticated biological organization. Organization invariably manifests FSC rather than successive random events (RSC) or low-informational self-ordering phenomena (OSC).,,,

    ,,,We repeat that a single incident of nontrivial algorithmic programming success achieved without selection for fitness at the decision-node programming level would falsify any of these null hypotheses. This renders each of these hypotheses scientifically testable. We offer the prediction that none of these four hypotheses will be falsified.
    http://www.tbiomed.com/content/2/1/29

    Programming of Life – Information – Shannon, Functional & Prescriptive – video
    https://www.youtube.com/watch?v=h3s1BXfZ-3w

    Programming of Life – Dr. Donald Johnson (the difference between Shannon Information and Prescriptive Information) – audio podcast
    http://www.idthefuture.com/201....._life.html

    But even using the ‘loose’ definition of Shannon information, instead of using the more precise definitions of functional information, (i.e. CSI, prescriptive, etc..), has insurmountable problems for Darwinists.

    The GS (genetic selection) Principle – David L. Abel – 2009
    Excerpt: Stunningly, information has been shown not to increase in the coding regions of DNA with evolution. ,,,
    What the above papers show is that not even variation of the duplication produces new information, not even Shannon “information.”
    http://www.bioscience.org/fbs/.....6/3426.pdf

    Moreover, using the Shannon metric for information (i.e. channel capacity), we find the first DNA code of life on earth had to be at least as complex as the current DNA code found in life is:

    Shannon Information – Channel Capacity – Perry Marshall – video
    http://www.metacafe.com/watch/5457552/

    “Because of Shannon channel capacity that previous (first) codon alphabet had to be at least as complex as the current codon alphabet (DNA code), otherwise transferring the information from the simpler alphabet into the current alphabet would have been mathematically impossible”
    Donald E. Johnson – Bioinformatics: The Information in Life

    “Biophysicist Hubert Yockey determined that natural selection would have to explore 1.40 x 10^70 different genetic codes to discover the optimal universal genetic code that is found in nature. The maximum amount of time available for it to originate is 6.3 x 10^15 seconds. Natural selection would have to evaluate roughly 10^55 codes per second to find the one that is optimal. Put simply, natural selection lacks the time necessary to find the optimal universal genetic code we find in nature.”
    (Fazale Rana, -The Cell’s Design – 2008 – page 177)

    The reason why a code is a all or nothing deal that cannot be evolved gradually is best summarized by Dawkins:

    Venter vs. Dawkins on the Tree of Life – and Another Dawkins Whopper – March 2011
    Excerpt:,,, But first, let’s look at the reason Dawkins gives for why the code must be universal:
    “The reason is interesting. Any mutation in the genetic code itself (as opposed to mutations in the genes that it encodes) would have an instantly catastrophic effect, not just in one place but throughout the whole organism. If any word in the 64-word dictionary changed its meaning, so that it came to specify a different amino acid, just about every protein in the body would instantaneously change, probably in many places along its length. Unlike an ordinary mutation…this would spell disaster.” (2009, p. 409-10)
    OK. Keep Dawkins’ claim of universality in mind, along with his argument for why the code must be universal, and then go here (linked site listing 23 variants of the genetic code).
    Simple counting question: does “one or two” equal 23? That’s the number of known variant genetic codes compiled by the National Center for Biotechnology Information. By any measure, Dawkins is off by an order of magnitude, times a factor of two.
    http://www.evolutionnews.org/2.....44681.html

    And this problem of multiple codes, that can’t be derived by gradual processes, has recently become much more acute for Darwinists:

    A glimpse into nature’s looking glass — to find the genetic code is reassigned: Stop codon varies widely – May 22, 2014
    Excerpt: While a few examples of organisms deviating from this canonical code had been serendipitously discovered before, these were widely thought of as very rare evolutionary oddities, absent from most places on Earth and representing a tiny fraction of species. Now, this paradigm has been challenged by the discovery of large numbers of exceptions from the canonical genetic code,,,
    Approximately 99% of all microbial species on Earth fall in this category, defying culture in the laboratory but profoundly influencing the most significant environmental processes from plant growth and health, to the carbon and other nutrient cycles on land and sea, and even climate processes.,,,
    “We were surprised to find that an unprecedented number of bacteria in the wild possess these codon reassignments, from “stop” to amino-acid encoding “sense,” up to 10 percent of the time in some environments,” said Rubin.
    Another observation the researchers made was that beyond bacteria, these reassignments were also happening in phage, viruses that attack bacterial cells.,,,
    The punch line, Rubin said, is that the dogma is wrong.
    “Phage apparently don’t really ‘care’ about the codon usage of the host.
    http://www.sciencedaily.com/re.....141422.htm

    supplemental notes:

    “In the last ten years, at least 20 different natural information codes were discovered in life, each operating to arbitrary conventions (not determined by law or physicality). Examples include protein address codes [Ber08B], acetylation codes [Kni06], RNA codes [Fai07], metabolic codes [Bru07], cytoskeleton codes [Gim08], histone codes [Jen01], and alternative splicing codes [Bar10].
    Donald E. Johnson – Programming of Life – pg.51 – 2010

    “Not only are there many different codes in the sequences, but they overlap, so that the same letters in a sequence may take part simultaneously in several different messages.”
    Edward N. Trifonov – 2010

  10. 10
    Neil Rickert says:

    Responding to UB’s comment.

    Sorry, but you are confused here.

    You recognize the “apple” because it’s specified; you know what those particular scribbles mean. Shannon is telling you that this process of selection at the receiver is irrelevant to the problem of engineering a communication channel that can carry any scribble – the one selected and all the others that are not, i.e. the noise. That is what the system has to be able to do in order to function.

    The actual engineering that Shannon mentions, mostly consists of specifications and of hardware that implements those specification.

    This is not the ID kind of specification, as in “Oh, it looks specified, so it just must be.” These are detailed engineering specification that you can find in libraries and probably on the internet.

    A varying electrical signal can be converted to a sequence of bits in a gazillion different ways. But, somehow, the Internet works because we always manage to get the bits that were intended. It works because of all of those engineering specifications, and because the equipment all follows those specifications both in generating the signal and in interpreting it.

  11. 11
    Eric Anderson says:

    GBDixon @7:

    Thanks for your thoughts.

    Are you suggesting that if we know, for example, that we are dealing with, say, a 4-bit digital code, that it is impossible to calculate the shannon metric without first knowing all of the possible messages that can be sent using that 4-bit digital code?

    I hope that is not what you meant by your first paragraph, because that is completely wrong.

    Shannon did more than determine channel capacity. He gave a fundamental definition of what constitutes information. This applies to all information. That is why his work is considered groundbreaking.

    No. He focused on one particular aspect of communication which is highly relevant to the transmission of information. That is why his work is considered groundbreaking.

    Again, Shannon acknowledged that the meaning and the semantics — what we generally think of as information if you were to ask anyone on the street, if you were to look the word up in the dictionary, if you were to look at the etymology of the word — is irrelevant to his calculation.

    In Eric’s fruit truck example there are more than one receivers and more than one set of information. To the woman who calculates truck gas consumption, the information is the weight of the load and the length of the route. To the person who loads the truck, the information is how much space is left. To the person who unloads the truck it is how much more is left to unload before I can take my coffee break. To the customer, it is what is wanted.

    Look, any analogy can be stretched too far. You are going off on tangents. The reason I worded the analogy the way I did is to focus on the carrying capacity, which is what the Shannon metric deals with. Don’t get off on irrelevant tangents about the driver, the coffee breaks for the dock workers, etc.

    Yes, there is lots of “information” about a physical system that can be gleaned by someone observing the system. That is separate from the information contained in the transmission itself. (This question about information being contained in a physical system has been dealt with in detail previously on this site.)

    Focus on (i) the size of the truck, (ii) the sizes of the different packets that can be carried on the truck, and (iii) the contents of the packets. Those are the key.

    We usually presume that the sender only sends the things the receiver (customer) wants and that something bad happens to the messages along the way. In biology, this is usually considered to be the DNA mutations that creep in and do nothing or cause problems.

    Yes, noise is an interesting issue, but it is easily included in our example if you want (spoilage in transit, box falling off the truck, someone stealing fruit from a box while the driver is parked at the coffee shop, etc.). Noise is an important point for transmission and one that Shannon spent a lot of time focusing on.

    However, I would note that most of the calculations of the Shannon metric that are done in the context of the design debate assume a noiseless channel. Indeed, we need to assume a noiseless channel initially to come up with a base calculation (as Shannon also did). Noise makes the question of transmission more challenging, but it is not typically relevant for the encoding aspect we are thinking about.

  12. 12
    Neil Rickert says:

    Responding to EA’s comment (#8):

    Well, which is it, the information or the channel capacity?

    Strictly speaking, the metric is the amount of information. The channel capacity is a rate (max amount of information per unit time).

    In which case we have then lost the value of the word “information” and we need to come up with a different word to describe what we originally used to think of as information.

    No, we have not lost the value of the word “information.”

    As Shannon has said, information can carry semantics. But, as defined by Shannon, the term “information” does not refer to the semantics.

    I understand that you want “information” to actually refer to the semantics. I used to think that way, and I probably posted comments to that effect on usenet, maybe 20 years ago. I have since come to recognize that I was mistaken. I now see that Shannon information, which does not refer to the semantics, is the most appropriate meaning for “information.” And, here, I am talking about what is appropriate for studying questions of human cognition. I’m not limiting myself to digital technology.

    The problem with semantics (or meaning), is that it cannot be specified. If you want to insist that “information” is a reference to semantics, then you must give up on the “specified” part of CSI. It is the nature of ordinary language, that meanings are inherently subjective and unspecifiable.

  13. 13
    bornagain77 says:

    A few related notes that may be of interest to the reader:

    Measuring the functional sequence complexity of proteins – Kirk K Durston, David KY Chiu, David L Abel and Jack T Trevors – 2007
    Excerpt: We have extended Shannon uncertainty by incorporating the data variable with a functionality variable. The resulting measured unit, which we call Functional bit (Fit), is calculated from the sequence data jointly with the defined functionality variable. To demonstrate the relevance to functional bioinformatics, a method to measure functional sequence complexity was developed and applied to 35 protein families.,,,
    http://www.tbiomed.com/content/4/1/47

    At the 17 minute mark of the following video, Winston Ewert speaks on how functional information is measured in proteins:

    Proposed Information Metric: Conditional Kolmogorov Complexity (Ewert) – July 2012 – video
    http://www.youtube.com/watch?v=fm3mm3ofAYU

    =============

    Conversations with William Dembski–The Thesis of Being as Communion (The Metaphysics of Information) – video
    https://www.youtube.com/watch?v=cYAsaU9IvnI

  14. 14
    Upright BiPed says:

    Good grief Neil.

    Let’s run this down:

    Eric made a comment about the specification inherent in information – the “aboutness” of information.

    You then turn around to object, saying “no” all Shannon information is specified, ‘noise isn’t considered Shannon information.’

    I then point out to you that your comments are in direct opposition to what Shannon himself is saying regarding the “aboutness” of information (it’s specification) – i.e. its “correlation according to some system with certain physical or conceptual entities”.

    So now you turn around and say the “specification” you’re talking about is about the system instead, not the information itself. Moreover, you want to pretend that when Shannon was discussing the selection of a message for its specificity (its “correlation according to some system with certain physical or conceptual entities”), he talking about the system specification as well. You are just deeply confused.

    Finally, after having bastardized what Shannon has said, you want to use the opportuinity to take a juvenile slap at ID for having an appropriate interest in the aspects of information that are “correlated according to some system with certain physical or conceptual entities”. Obviously, you take a shot at ID on an issue you clearly do not understand.

    Give it a rest.

  15. 15
    GBDixon says:

    Hi Eric,

    In your first example of a four bit code:
    The sender and receiver decide 1001 and 0110 are valid messages. This is the information. The sender only sends one of these two messages when she sends.

    The channel possibly corrupts the messages into any of the other possible fourteen combinations of four bits, or leaves the messages the same. the fourteen combinations are not found in the message list and are NOT information.

    These invalid messages plus the valid messages are the set of all possible four bit combinations that can be received and with these two parameters we can calculate the entropy of the information. Without an agreed-upon message list and the number of invalid messages that are possible, entropy cannot be calculated.

    Note that we could have chosen any combination of four bits as a valid message. This is what Shannon means when he says the content of the messages do not matter. We may attach meaning to a message (“launch the water balloon”) but as fundamental information, only what is in the valid message list is information according to Shannon.

    I attach semantics and go off on tangents to illustrate information depends on sender and receiver context. Here, the 1001 and 0110 messages are meaningless to us, but we can assert they are information because they are in the valid message list. They presumably have pithy meaning to both the sender and receiver but we as channel engineers are not privy to that meaning and don’t care what it is: only that 1001 and 0110 are valid messages and the rest are not (we discard invalid messages and ask for a resend or do something else).

    Attaching semantics to valid messages does not alter any of the calculations or what constitutes a valid message but if we wish to know why 1001 is a valid message, we must consult the sender or receiver and find out what the message means. We have generalized the concept of information, as Shannon did (Shannon actually built on the work of Hartley).

    Shannon did indeed generalize and formalize the concept of information. I think you may be stuck on Shannon’s channel capacity theorem, but he generated two fundamental theorems: the other is called the “source coding theorem” and deals with what information is.

    Many very smart people had trouble with Shannon’s concepts at first. I recommend John R. Pierce “An Introduction to Information Theory” for a good explanation. It is cheap at Amazon.

    Eric says:
    Focus on (i) the size of the truck, (ii) the sizes of the different packets that can be carried on the truck, and (iii) the contents of the packets. Those are the key.

    I agree with this as it relates to channel capacity, Eric, but aren’t we mainly talking about what constitutes information and not how to send it?

    Where we appear to disagree is the idea I assert that the receiver (and usually the sender) determine what the information is and what it means. Regardless of what they choose, the information can be distilled into a set of messages that are generic in nature and are sent across a possibly noisy channel to be received, checked for validity, decoded and interpreted by the receiver (again, in cells the receivers are the molecular machines that interpret the DNA sequences). Information cannot be defined without a receiver to interpret it, and information changes with receiver context.

  16. 16
    Joe says:

    Neil- a definition of a word specifies that words meaning. And the entire world uses the word “information” in a way that information = meaning. Textbooks are full of meaningful information.

    Shannon did not care about meaning because, guess what, the machines that transmit and receive the signal don’t care about it.

  17. 17
    gpuccio says:

    GBDixon:

    Thank you for your clear summary of some basic concepts about Shannon’s theory.

    I would like to mention here how the concepts of Shannon are applied in Durston’s model. There, the random state of a protein sequence of a certain length is considered as the highest uncertainly (highest H). The contraints given by the functional state (derived by the alignment of the known variants of that protein in the course of evolution) determine a reduction of uncertainty, which corresponds to the functional information in the protein sequence.

    In general, for a protein sequence, the total number of sequences which exhibit the molecular function is the “list” of meaningful “messages”. Their meaning is their functional activity, and can be defined independently. In itself, the function has nothing to do with the calculation, except for its constraints on the sequence and the consequent reduction of uncertainty. So, an AA position which always remains the same will give a reduction of uncertainty of a little more than 4 bits, while an AA position which can assume randomly any value will give no reduction of uncertainty (0 functional information). All intermediate conditions can be found. The sum of the functional information at each position is the global functional information of that protein.

  18. 18
    Neil Rickert says:

    Replying to UB’ comment (#14):

    Eric made a comment about the specification inherent in information – the “aboutness” of information.

    Aboutness (intentionality) is not specification.

    I then point out to you that your comments are in direct opposition to what Shannon himself is saying regarding the “aboutness” of information (it’s specification) – i.e. its “correlation according to some system with certain physical or conceptual entities”.

    Again, aboutness is not specification.

    I don’t recall Shannon mentioning “aboutness”, though he did mention meaning and semantics.

    A specification ought to be specific. Meaning is never specific — it is always subjective and dependent on subjective interpretation.

    … when Shannon was discussing the selection of a message for its specificity (its “correlation according to some system with certain physical or conceptual entities”) …

    Specification and correlation are not the same thing at all.

    When a topic claims to be about Shannon information, then it ought to be about Shannon information. Meaning and semantics, in the ordinary language sense, are not any part of Shannon information. Typically, Shannon information is a sequence of symbols. Specification, in the context of Shannon information, can only mean specification of symbols.

    If you wanted to discuss something other than Shannon information, this thread seems to be the wrong place for that discussion.

  19. 19
    Neil Rickert says:

    Responding to Joe (#16):

    Neil- a definition of a word specifies that words meaning.

    A definition of a word is an entry into a circular chain of dictionary lookups which never get you to a specification of meaning.

  20. 20
    Joe says:

    Neil Rickert:

    A definition of a word is an entry into a circular chain of dictionary lookups which never get you to a specification of meaning.

    Yes, Neil, we already understand that you choose obfuscation rather than education.

    If we didn’t define words, ie if words were not specified, communication would be impossible. Definitions are word specifications.

  21. 21
    Joe says:

    The word information in this theory is used in a special mathematical sense that must not be confused with its ordinary usage. In particular, information must not be confused with meaning.– Warren Weaver, one of Shannon’s collaborators

    And from data:

    Data as an abstract concept can be viewed as the lowest level of abstraction, from which information and then knowledge are derived.

    …..

    Beynon-Davies uses the concept of a sign to distinguish between data and information; data are symbols while information occurs when the symbols are used to refer to something.

  22. 22
    Neil Rickert says:

    If we didn’t define words, ie if words were not specified, communication would be impossible. Definitions are word specifications.

    Successful communication was going on thousands of years before there were definitions or dictionaries or written language.

    Bird manage to communicate with their bird songs. Where are the definitions that you claim would be needed?

  23. 23
    Upright BiPed says:

    If you wanted to discuss something other than Shannon information, this thread seems to be the wrong place for that discussion.

    Neil, you made a claim about Shannon information that is patently wrong. You were corrected. Shannon information does not consider any specification in the communication channel, consequently, unspecified noise in the communuication channel is included in Shannon information – just as Claude Shannon states in the second paragraph of his paper.

  24. 24
    Eric Anderson says:

    Neil:

    Let’s not get all hung up on a definitional battle. I’ve already said that if someone wants to use the term “Shannon information,” fine. I’m not going to change that unfortunate term this late in the game.

    But they need to understand that when they use the term “Shannon information” they are not talking about meaning, or content, or specification, or function. Shannon himself makes this quite clear.

    We could call the shannon metric something like “statistical information” if people insist on calling it “information” at all. And we could call the other stuff “substantive information.” The particular definition doesn’t matter. As long as people are clear that there is a real, fundamental, substantive distinction between the two.

  25. 25
    GBDixon says:

    Gpuccio
    The protein model seems very reasonable. Thank you.

  26. 26
    Eric Anderson says:

    Neil:

    Take the following string:

    ETTTBSNHOBOORTUEHTISEEATQTNOIO

    What is the Shannon information?

    This is not any kind of trick question. I’m sincerely trying to make sure we aren’t talking past each other and that I understand your point. Assume the foregoing is based on a 26-letter alphabet, all caps, no other symbols.

    I think we’re on the same page that meaning and content are not part of Shannon information, so I want to see if we can get on the same page as to what is Shannon information.

    Thanks,

  27. 27
    Joe says:

    Neil Rickert:

    Successful communication was going on thousands of years before there were definitions or dictionaries or written language.

    The context was communication using words, Neil. But thanks for proving you prefer obfuscation over education.

    Bird manage to communicate with their bird songs.

    How do you know they are communicating?

  28. 28
    johnp says:

    Neil Rickart at #12 said:

    The problem with semantics (or meaning), is that it cannot be specified. If you want to insist that “information” is a reference to semantics, then you must give up on the “specified” part of CSI. It is the nature of ordinary language, that meanings are inherently subjective and unspecifiable.

    In other words he agrees with the OP exactly as written. Thanks for the confirmation Neil!

  29. 29
    Neil Rickert says:

    Shannon information does not consider any specification in the communication channel, consequently, unspecified noise in the communuication channel is included in Shannon information – just as Claude Shannon states in the second paragraph of his paper.

    This is plainly wrong.

    Shannon was concerned about noise, and does not count noise as information.

    Suppose I have a random generator, set to generate random strings. I take the output and put that up on a web page.

    If you read that web page, you will that particular string, exactly as specified by my random generator. You won’t see a different string from what was specified. Maybe it looks like noise to you, but it is what the random generator specified. And the theory is concerned with transmitting that correctly (i.e. as specified).

    There’s a lot of electrical noise around. But you still see the string that my random generator produced (and thereby specified). You do not see any sign of the noise.

  30. 30
    Neil Rickert says:

    The context was communication using words, Neil.

    People were succesfully communicationg with words thousands of years before there were written languages or definitions or dictionaries.

  31. 31
    Neil Rickert says:

    Take the following string:

    ETTTBSNHOBOORTUEHTISEEATQTNOIO

    What is the Shannon information?

    The string itself is the information.

    If you intended to ask about the amount of information (the metric), I guess that’s about 141 bits (the number of letters, times the log of 26 to the base 2).

  32. 32
    Joe says:

    Neil Rickert:

    People were succesfully communicationg with words thousands of years before there were written languages or definitions or dictionaries.

    So those words had no meaning then. What, exactly, were they communicating with those meaningless words? And if words don’t have any meaning how can you tell if they are words?

    We are right back to:
    If we didn’t define words, ie if words were not specified, communication would be impossible. Definitions are word specifications.

    The definitions don’t have to be written down, Neil.

  33. 33
    Axel says:

    Not only do words have to be defined, but ball-park definitions, at the very least, have to be agreed.

    As a matter of fact, agreement/usage is paramount, much to the chagrin of the Academie Francaise. So, you could scarcely be more comprehensively mistaken, Neil.

  34. 34
    Upright BiPed says:

    – facepalm –

    Neil now wants to hook up a random generator to the input channel of the system in order to “specify” randomness to the output. By doing this, he has concocted a scenario where he can save his claim that “all Shannon information is specified”.

    .
    (I believe I’ll move along now)

  35. 35
    groovamos says:

    I’m really curious at what level the OP understands the subtle implications of information theory, me thinking probably he doesn’t. Conversely, it seems the OP thinks that information theory is an attempt to conform to some function that he thinks it should but but somehow comes up short in its founder’s aspirations to achieve, maybe because of the misunderstanding of the subtleties.

    For example, because of the OP’s little scenario it seems to me that he thinks there should be some type of quality factor attached to information, as if an artistic work could somehow be measured and shoveled in there. Or somehow the qualities of decision making by the young manager on the job. The original OP seems also to be confused by what Shannon was referring to as capacity. In a channel operating near BIT RATE capacity, the error-correction codes are embedded such that the message can be recovered WITHOUT ERROR due to the employment of the error-correction code (redundant information). It is the amazing result of the Shannon- Hartley theorem that it takes into account the success of any error-correction code and is one of those mysterious results from applied math that the theorem MUST take into account the action of error-correction whether or not the originators of the theorem intended this to be the case. And if you read the 1948 paper as I have, Shannon discusses this in entertaining terms. All communications will produce errors, but the amazing thing is that error correction can reduce the errors to zero as long as the recovered information bit rate does not exceed that specified by the Shannon-Hartley theorem, irregardless of the efficiency of the error-correction code employed. (efficiency measured as the numbers of errors corrected per second per additional bandwidth required for the redundant bits for the error correction code)

  36. 36
    Mung says:

    1.) Shannon information is information.

    2.) Shannon was completely correct to not refer to it as a measurement or a metric, because what he explicitly denies is that it tells you whether or not you are measuring information!

    3.) People who deny #1 are mistaken. They fail to understand what Shannon information is information about. It is explicitly not about measuring information!

    4.) People who do not understand #2 and #3 often mistakenly claim that Shannon information demonstrates that information can be meaningless.

    We get our fair share of both here at UD. Allen MacNeill and Elizabeth Liddle spring to mind.

    But meaningless information? Really? How does anyone get that from Shannon’s theory? Please. Speak up.

    So, Eric, thanks again for treading where angels fear 🙂

    Questiosn for all to ponder:

    Q1: What is Shannon Information?

    Q2: What is Shannon Information about?

    Q3: Is Shannon Information independent of the information content (or lack thereof) of the message?

    Q4: Does Shannon Information tell us whether the message is meaningful or not?

    Q5: If Shannon Information cannot tell us whether the message is meaningful (or not) can it therefore be deduced from Shannon’s theory that information can be meaningless?

    Q5: What would “meaningless Shannon Information” look like?

  37. 37
    Joe says:

    Mung:

    But meaningless information? Really? How does anyone get that from Shannon’s theory? Please. Speak up.

    Warren Weaver, one of Shannon’s collaborators:

    The word information in this theory is used in a special mathematical sense that must not be confused with its ordinary usage. In particular, information must not be confused with meaning.

    What is information without meaning if not meaningless information?

  38. 38
    Mung says:

    Don’t mean to pick on you Neil, because I haven’t yet read all your comments in the current thread, but you make some fundamental errors which will hopefully be instructive.

    The string itself is the information.

    No, it isn’t. Shannon’s theory does not tell you whether this message constitutes information or not.

    If you intended to ask about the amount of information (the metric), I guess that’s about 141 bits (the number of letters, times the log of 26 to the base 2).

    Is that assuming an alphabet of 26 symbols with each having an equal probability? IOW, not English?

    I say assuming, because how did you know? Did Shannon’s theory tell you?

    Eric Anderson:

    Take the following string:

    ETTTBSNHOBOORTUEHTISEEATQTNOIO

    What is the Shannon information?

    That may not be a “trick” question, but it is a meaningless question. In the English language there are a specific number of letters and certain letters appear more frequently than others. Is that a message in the English language? I think not.

    So how then are we able to reduce the uncertainty?

  39. 39
    Eric Anderson says:

    Neil @31:

    The string itself is the information.

    Thanks. Let me make sure I’m understanding you.

    With a string, say, ETTTBSNHOBOORTUEHTISEEATQTNOIO, you are saying that the string itself is “Shannon information.”

    Presumably, then, the string HTISEEATQTNOIOETTTBSNHOBOORTUE would also be “Shannon information.”

    And the string HTISEEATQSNHOBOORTUETNOIOETTTB would also constitute “Shannon information,” and so on.

    In other words, under your definition any string of characters constitutes “Shannon information.” And a single letter and a whole page of characters or a whole book also each constitute “Shannon information.”

    That being the case, perhaps you can clarify for me what the difference is between Shannon information and strings of characters? Under your definition they seem to be equivalent. If you are right, then we can say:

    Shannon information = string of characters

    Which is to say, if we have a string of characters, then the Shannon information concept tells us that we have . . . a string of characters.

    What does the term “Shannon information” bring to the table in your definition, if it simply is another way of saying that it is “the string itself”?

  40. 40
    Eric Anderson says:

    Mung:

    Thanks for your comments. I definitely want your input, as you have been very vocal about this for a long time. I do trust that after we’ve had a chance to mull on your questions for a day that you’ll treat us to an actual explanation and not just let us flounder at sea while you ask provoking questions. 🙂

    That may not be a “trick” question, but it is a meaningless question.

    It is not meaningless at all. I’m trying to step back to square one to understand what Neil is referring to when he talks about “Shannon information.” The example is very much on point. Yes, I could have said the letters had equal probability if we wanted to make the calculation simpler. That impact on the calculation is not directly important, however, to the question I posed. Hopefully, my follow up @39 will help flesh out precisely what Neil views as “Shannon information” so that I can understand it and be on the same page.

  41. 41
    Mung says:

    Joe:

    What is information without meaning if not meaningless information?

    Give me an example of meaningless information.

    The word information in this theory is used in a special mathematical sense that must not be confused with its ordinary usage. In particular, information must not be confused with meaning.

    So when he says information in the second sentence you don’t think he means information [as used] in this theory … in a special mathematical sense?

    So why should information have a special sense in this theory but be nonsense elsewhere?

    If information can be meaningless, why is Shannon Information exempt? Is Shannon Information not information? Do you think he means Shannon Information can be meaningless?

    All Weaver is saying is what I said previously. Shannon Information is mathematical, it doesn’t tell you whether whether or not the message is meaningful. It does not logically follow that information can be meaningless.

    In fact, it is not logically possible for information to be meaningless.

    But to the specific challenge I raised, which you did not address. How does one get “meaningless information?” from Shannon’s theory. Given that it’s a mathematical theory of communication, I expect to see some maths.

    A fundamental, but a somehow forgotten fact, is that information is always information about something.

    The Mathematical Theory of Information

  42. 42
    Mung says:

    hi Eric,

    When I said the question was meaningless, I meant that the question did not have an answer according to Shannon’s theory.

    It’s like asking what is the Shannon information of the sky. Like asking whether a red sky has more or less Shannon information than a blue sky.

    Yes, Shannon Information is a measure, but it’s a measure of probabilities, not a measure of information in the classical sense.

    Let me try to give an example:

    Say we could create a symbol generator that could generate an infinite number of different symbols each with the same probability. Could Shannon theory be used to measure the information in bits?

    Say we could create a symbol generator that could generate a finite number of different symbols but with no predictable probability for any given symbol. Could Shannon theory be used to measure the information in bits?

    If you have a “coin tossing” machine and the two symbols it generates are H and T respectively and they are equiprobable, then when you see an H you can say your uncertainty has been reduced by a certain amount, and we can call this “information.”

    But if we randomly change the probability of the T or the H, or if we randomly insert other symbols, then how do we measure the probability, or the reduction in uncertainty, or the “Shannon Information”?

    Cheers

  43. 43
    Mung says:

    Nota bene:

    I drafted this post more than two months ago

    The Shannon Information has degraded accordingly!

  44. 44
    Mung says:

    Neil Rickert:

    Just to be clear, the metric is not the information.

    The measure is not that which is being measured?

    I could not agree more. Confusing the two is a source of much confusion.

    Is it not also the case that if the measure is “an amount of information” it does not logically follow that it is information that is being measured?

    Neil Rickert

    Just to be clear, the metric is not the information.

    It does not follow that because a measurement can be taken in “amounts of information” that what is being measured is information.

    It also does not follow that because a measurement can be taken in “amounts of information” that what is being measured is meaningless information.

    Neil Rickert

    The metric is a way of measuring the amount of information (or the channel capacity).

    The amount of information of what?

    All that’s being “measured” is the capacity to process symbols. The symbols may or may not convey meaning. Whether or not they convey meaning is irrelevant to the mathematical problem. From this, it does not logically follow that there is or can be such a thing as “meaningless information.”

  45. 45
    Mung says:

    GBDixon:

    The set of valid messages among the total of all possible messages that can be received defines the information. Channel capacity has no meaning without this message list and information entropy and the channel capacity cannot be calculated without knowing the number of valid messages among the number of possible messages.

  46. 46
    Mung says:

    Neil Rickert:

    This is not the ID kind of specification, as in “Oh, it looks specified, so it just must be.” These are detailed engineering specification that you can find in libraries and probably on the internet.

    Right. These are real specifications

    But they have nothing to do with “the ID kind of specification.” Right.

  47. 47
    Mung says:

    Eric:

    Shannon information = string of characters

    This is what Elizabeth Liddle thought. Toss a coin x number of times you get y bits of “Shannon information.”

    The coin tosses were meaningless. When asked what they were about she had no answer. But because y bits of Shannon Information could be calculated it somehow meant that information could be meaningless.

    Unfortunately for ID, that is also the view presented by Stephen Meyer in Signature in the Cell.

  48. 48
    Mung says:

    Eric:

    The Analogy

    The fruit truck is obviously but a simple and fun analogy. However, it does, I believe, help newcomers get a feel for what Shannon can do (analyze maximum carrying capacity of a delivery channel) and what Shannon cannot do (analyze, confirm, understand or quantify the underlying substance).

    Are we then in agreement that Shannon theory cannot tell us what is information and what is not information? That Shannon theory cannot tell us which messages are meaningful and which are not meaningful?

    Because Shannon theory lacks this capability, does it in any way logically follow from Shannon theory that information can be meaningless?

    Say the fruit truck confused Avocado with Almond. IF the symbol “A” could mean either one what are the implications?

  49. 49
    Eric Anderson says:

    Mung @41-47:

    Whoa, slow down there cowboy! Let Neil get a word in edge-wise! 🙂 I know you’re already fired up about this!

    @47:

    Unfortunately for ID, that is also the view presented by Stephen Meyer in Signature in the Cell.

    Do you have a cite for this? I could maybe look it up, but I’m too lazy and I might not find the actual passage that you are referring to, so if you have a quote handy, that would be helpful.

    If what Meyer is saying is that a string of characters can be meaningless, then I’d have to agree.

    If what you’re saying is that a string of characters (given relevant parameters, of course) can contain a certain number of bits of Shannon information and that, therefore, the Shannon information is “meaningful” because it is “about” something, then I’d say that is trivially true . . . and also wholly uninteresting for purposes of ID, which is no doubt what Meyer was focusing on.

    Anyway, if you have the quote handy, that would be great.

  50. 50
    Neil Rickert says:

    That being the case, perhaps you can clarify for me what the difference is between Shannon information and strings of characters?

    Strictly speaking, it is Shannon information if it is part of a communication system or an information processing system.

    Marks on paper that just happen to look like letters would not count. This is why it is reasonable to say that Shannon information is always specified (by being entered into a communication system).

  51. 51
    Eric Anderson says:

    Hmmm . . . Slipped in a #48 while I was typing, eh?

    Are we then in agreement that Shannon theory cannot tell us what is information and what is not information?

    Well, that depends on what we mean by information. I’m inclined to think that information has to be meaningful, but other folks (as we have seen), argue that any old string of letters can be information. That is part of the definitional confusion that results from the term “Shannon information.”

    That Shannon theory cannot tell us which messages are meaningful and which are not meaningful?

    Quite true. Shannon himself said as much. Indeed, he used the word “irrelevant” when talking about the meaning of a message in the context of his theory.

    Because Shannon theory lacks this capability, does it in any way logically follow from Shannon theory that information can be meaningless?

    Again, this is a definitional issue. Shannon theory simply cannot say whether we are dealing with meaning or not.

    So if information necessarily has meaning, as you appear to be arguing (and with which I would provisionally be inclined to agree), then — by definition — Shannon theory cannot tell us whether we are dealing with information or not. At least not in the underlying substance.

    Thus the so-called “Shannon information” must be about something entirely different than the substance of the message, the substance of the string, the substance of the communication. The metric is measuring something separate from and apart from the underlying information.

    That is part of my whole point.

  52. 52
    Neil Rickert says:

    Mung (#38):

    http://www.uncommondescent.com.....ent-505811

    Strictly speaking, that is correct. It is a mathematical theory. One chooses whether to use the theory to model an actual communication system. However, people find it more convenient to say “x is Shannon information” than “x can be modeled as Shannon information.”

  53. 53
    Eric Anderson says:

    Neil @50:

    Thanks.

    This is why it is reasonable to say that Shannon information is always specified (by being entered into a communication system).

    Just to be clear, are you saying that something entered into a communication system has to be a valid communication (i.e., meaningful, understandable, purposeful, or whatever similar term we want to use here)?

    Such underlying meaning would indeed be a specification.

    Or are you arguing that the simple act of entering a string into the system (whether or not that string has any underlying meaning or substance) makes that string a specification?

  54. 54
    Neil Rickert says:

    Eric (#51):

    Well, that depends on what we mean by information.

    This may be the most important point of the whole discussion. The word “information” is used in multiple conflicting ways. Unfortunately, people talk is if information were an objective or metaphysical entity. We would do better to understand it as an attribution appropriate for a particular use.

  55. 55
    Neil Rickert says:

    Just to be clear, are you saying that something entered into a communication system has to be a valid communication (i.e., meaningful, understandable, purposeful, or whatever similar term we want to use here)?

    No. It is best to leave those ideas out of the issue.

    A video camera transmitting data could still count as a communication system, even if it is entirely mechanical and no part of the system has a capacity for meaning or understanding.

    You can look at it as in my reply to Mung. What makes something information, is our choice to use Shannon’s theory to mathematically model it. Your string of letters is Shannon information, in the sense that I took you as communicating that string in a way that can be modeled as Shannon information. However, in completely objective metaphysical terms, your string of letters is ink marks on paper or illuminated dots on a screen.

    The reason I like Shannon information, is that it gives us that stark picture that nothing actually is information. Rather, we count something as information based on how we use it.

    And sure, we have people claiming that the Universe itself is just information. Whatever it is that they mean by “information”, it isn’t anything that I can make sense of.

  56. 56
    Joe says:

    It’s best to leave whatever Neil sez out of the discussion as it has become very obvious that Neil doesn’t know jack about information.

    BTW marks on a paper that look like letters could very well be information. As I said, Neil is the wrong person to discuss information with as he thinks communication can take -place with meaningless words- whatever those are.

  57. 57
    GBDixon says:

    Hi all,

    This will be my last comment. I have irritated our host and come off as a know-it-all. This is important to me, I guess. I think that a lot of confusion, argument and new vocabulary could be avoided if ID advocates knew information theory a little better. Perhaps this summary may help a little:

    Shannon divided the information transmission problem into two pieces: the encode/decode problem and the transmit/receive problem. His famous channel capacity theorem is the solution to the second problem and is the domain our host and most others have been working in.

    But the nature of information, what it is, and how to best represent it is the subject of the encode/decode problem. Here, information is defined, mainly by the receiver, and meaning is the most important aspect of the problem.

    In his encode/decode theorem, Shannon showed how a message with redundancy (an example of redundancy is how u nearly always follows q in English) could be encoded into the minimum-sized message possible (“pure” information if you will). We are all familiar with the technique: zip and tar files reduce size by removing redundancy. In this first problem domain information is simply what informs the receiver. Nothing more or less.

    Once the information is reduced to its smallest generic form, it is prepared to be sent through the channel and we enter the second problem domain. Here, a special form of redundancy (error correction codes) are added to the generic messages and they are transmitted. In this domain the meaning of the messages is irrelevant, thus the comments to this effect.

    But once the messages are received they are decoded and meaning is once again assigned to each message. We are back into the domain where meaning matters.

    It is as simple as that: information is the stuff the receiver sees as information. Complex patterns in mud do not inform us much…they don’t contain much information. But we have learned a great deal from DNA. It is tremendously information-rich to both us and the molecular machines that process it.

    We are privileged to live in a time when the Shannon channel capacity has been practically reached. We have technology that is fast and cheap enough to transmit information error-free at essentially the maximum theoretical rate.

    I believe the entire universe was set up so we are able to figure out, in every detail, exactly how it was all done. A magnificent classroom or lab, if you will. What marvelous discoveries await us!

  58. 58
    Eric Anderson says:

    information [in-fer-mey-shun] noun

    1. knowledge acquired through experience or study
    2. knowledge of specific and timely events; news
    3. the act of informing or the condition of being informed
    4. in computing: (a) the meaning given to data by the way in which it is interpreted; (b) another word for data

    Synonyms: data, facts, intelligence, advice.

    —–

    Neil suggests @55 that it is best to leave things like understanding and purposeful informing and meaning “out of the issue.”

    In other words, if we strip the word “information” of everything that makes it information, if we strip it of all ordinary meaning (no pun intended), then we can use the word to mean something else than what is normally meant. Shoot, if we’re going that far, why not just use the word “dogs” instead? We could strip it of all its normal meaning and say that we are now dealing with “Shannon dogs.” Just as rational.

    What makes something information, is our choice to use Shannon’s theory to mathematically model it.

    Sorry. Shannon was a great guy, but he doesn’t get to completely strip the word “information” of everything that the word has meant for hundreds of years and say, in effect, “anything that can be modeled with my theory is information, everything else isn’t.” Furthermore, Shannon never made any such claim.

    Again, I don’t begrudge anyone using the term “Shannon information” as long as they are clear what they are talking about. So far, it is wholly unclear what you mean.

    The reason I like Shannon information, is that it gives us that stark picture that nothing actually is information.

    Wait a minute. Any string, every string, can be modeled with the Shannon metric if we know the relevant parameters. Previously you said it was all information. Now you’re saying nothing is?

    Rather, we count something as information based on how we use it.

    Oh, I see. We count something as information if it is usable to convey something of substance, if it has meaning, if it informs. I certainly agree with that. And it takes us right back to the old dictionary definition.

    —–

    And sure, we have people claiming that the Universe itself is just information. Whatever it is that they mean by “information”, it isn’t anything that I can make sense of.

    I think we probably have the same uneasy skepticism here. Not real clear to me either, though I do understand the general thrust of information being at the heart of something or being the source of something.

  59. 59
    Joe says:

    GBDixon- Thank you for your posts- You haven’t irritated me and that says quite a bit because when it comes to information this information technologist is a know-it-all who would have been irritated had you posted nonsense. 😉

  60. 60
    Eric Anderson says:

    BTW, Neil, getting back to the question @53:

    Since you answered ‘no’ to the meaning option, I take it you are saying that the simple act of something being entered in a communication system makes it a “specification.”

    We could, I suppose, refer to the mere sequence of any string as a “specification”, which is really no more substantive than saying that the sequence is what it is. In other words, we would be saying that the sequence equals the specification and we could just use the word “sequence” instead of “specification”.

    Yes, we can refer to the sequence of a string as its “specification,” and that might make sense as a shorthand in some circumstances. That is a purely descriptive sense, and (as Shannon mentioned) is devoid of any concept of substance or meaning.

    But we need to keep in mind that this is different from the “specification” concept that design advocates talk about. In this latter case we are very much concerned with substance and meaning and purpose and function.

    —–

    This all really underscores the broader point of my post. Namely, the concept of Shannon “information” is not really helpful or relevant to the key issues of specified information and design. Shannon information is helpful and relevant only for the base identification of “complexity” in certain circumstances. So it can help with the “C” of “CSI”, but is not really useful for thinking about the “S” or even the “I”.

  61. 61
    groovamos says:

    You know really, you guys, at least some of you, I think you miss something really fundamental. The usefulness of information can only be surmised by a mind. Just because one of you throws out a string of characters and denies that it is information, that denial itself can be determined to be information, or denied to be information.

    For example the length of that string might be an indicator in how much of a hurry was the contributor. So when Eric A typed out his two strings above, he thinks they mean nothing. But I have a mind, strangely enough, and I can tell that there was a high probability his caps lock key was on during both of his strings. This is information. Also, since I can see that there were no spaces and no significant wording to his strings, I can discern something of his intent, and his intent was to post gibberish for part of the post and to use it for didactic purposes. Especially since he did not reveal in the post any real world process by which the strings were created, even when ignoring all of his philosophical points. This is information. He did not type any punctuation characters, and so I can tell that he was biased against a significant subset of the ASCII code, admittedly I don’t know why or if it was just a temporary bias, but that is more information.

    And actually Eric has never even said anything about why, when he types a meaningless string, why we are all confident that what he keyed is what we all see on our screens, because none of us can check for spelling on those strings. And so that tells me that Eric is confident that all of his keyed input appears with no errors on all of our screens, without speculating on why that is, after reaching each of us via a unique set of channels. And that his “meaningless” strings did transfer error-free and did not appear with a couple of non-trivial words to one or two of us because of errors. Eric’s confidence in that error-free transmission is more information about him, but I believe there is little probability he understands why it is error-free and that is information.

  62. 62
    Eric Anderson says:

    GBDixon @57:

    This will be my last comment. I have irritated our host and come off as a know-it-all.

    I hope you’re not referring to me, as I certainly haven’t been irritated by your comments. I appreciate your thoughts.

    But once the messages are received they are decoded and meaning is once again assigned to each message. We are back into the domain where meaning matters.

    Meaning definitely matters to the normal concept of information, and I think your breaking down the communication process is helpful.

    A separate question is whether there is information independent of the recipient. We have discussed this in some detail previously and, I think, concluded that information arises at the recipient side — by definition prior to the time it is encoded.

    Later, if there is a recipient who can receive and understand the information that has been encoded, great. If not, it doesn’t necessarily mean that the information doesn’t exist. It is still there and could be discovered later.

    For example, the information encoded in DNA was there — objectively functioning and doing what it was supposed to do — for millennia before the genetic code was discovered.

  63. 63
    jerry says:

    GBDixon,

    I hope you come back. Your explanation at 57 was one of the best I have seen on this site. I usually tune out when someone discusses Shannon information because it is usually obtuse and then there is the contentious bunch who always want to create noise and prevent communication.

    So reconsider. Your posts have been a breath of fresh air.

  64. 64
    Eric Anderson says:

    groovamos @61:

    With respect, you are getting caught up in irrelevancies.

    As an intelligent being, are there things you can ascertain or discover or infer about some object in the real world (whether a rock at the seashore or a string of characters on your screen)? Sure. Lots of things. You can infer that I wrote my string with Caps Lock on, on a computer, that it was transmitted without error, etc. Those are things you can ascertain — pieces of information you can discover — about the string.

    All of that is 100% irrelevant to the question that was posed: Does the sequence of characters itself contain information?

    We have addressed in detail previously on this forum the difference between (i) an observer being able to discover information about an object (what you are focusing on), and (ii) representative information encoded in or contained in the object.

    It is critical to distinguish between these two. Your comments are exactly on point for (i). But they are completely irrelevant to what we were discussing, which is (ii).

  65. 65
    Eric Anderson says:

    jerry @63:

    . . . create noise and prevent communication.

    LOL! Great use of Shannon terminology! 🙂

  66. 66
    groovamos says:

    Eric: I think it not irrelevant at all to point out that it requires a mind to determine relevance, but that is my opinion, maybe not yours. You intended for your string to be irrelevant. It is obvious that it was your intent, and whether or not it is relevant what your intent was to the study of statistical communications, is really a matter of personal opinion, which requires a mind.

    An anemometer connected to a telemetry system sends back what to you would likely look like “random” bit sequences. With correct knowledge of the formatting and coding ( a big chunk of information in itself) the bit stream can be converted in to a sequence of numbers in “real” time, possibly very useful information. A researcher tries to fit the numbers to a statistical curve and concludes that they indicate a Rayleigh probability density for the velocity. This might invoke from you a “so what” response. If you had another piece (a well-known one) of information in your mind, you might instead say “Oh the wind speed with direction using any orthogonal coordinate set is a joint Gaussian probability with equal variances”.

    So at different point along the communication/information chain your ability to make sense of an otherwise seemingly meaningless bit stream requires more information. How important the information is at any point of the analysis is obviously a matter of opinion or mind.

  67. 67
    GBDixon says:

    Eric@62,

    Thank you, you are very gracious.

    We have many situations regarding information, don’t we? Where there is a sender and receiver, they together establish what is information by a contract usually called a protocol. Our Ethernet, cell phones, etc. all operate under these various protocols.

    You touch on what might be called latent information: we learn nothing from it at present, but we know it is there and if we only knew a little more it would become true information to us. The Rosetta stone is an example, I think. Here we had all these ancient texts…we knew they had valuable information but couldn’t decipher it. After the Rosetta stone was discovered that information became available to us. Here was a situation where there were senders and receivers but no contract. We had to figure out the senders’ protocols on our own.

    The cell is similar. With each new discovery more information is unlocked. We know there is much, much more we do not yet understand. Because the sender is unknown to us, we are left to our own devices to figure out the protocols. I too believe latent information exists, that it is not true information to us simply because we do not know what it means, but that it will someday reveal many things. The ability of the cell to preserve seemingly worthless DNA sequences over eons hints that there are buried treasures we have yet to discover.

  68. 68
    Upright BiPed says:

    Is anyone else smiling. No? Okay…

    🙂

  69. 69
    Andre says:

    Fantastic discussion.

    I’ve hammered this home on numerous occasions…. the encoder/decoder problem….

    And this alone makes Darwinian evolution the poppycock it is…..

  70. 70
    Neil Rickert says:

    Eric (#60):

    We could, I suppose, refer to the mere sequence of any string as a “specification”, which is really no more substantive than saying that the sequence is what it is. In other words, we would be saying that the sequence equals the specification and we could just use the word “sequence” instead of “specification”.

    I’m not quite sure of the point that you are raising there.

    When you earlier asked about a string of letters, you called them letters. You did not call them pencil marks or illuminated dots on the screen. By calling them letters, you identified them as symbols rather than marks. And symbols, at least as I look at them, are abstract intentional objects rather than physical objects.

    I’m inclined to say that there is already some specification going on when we take something to be a symbol.

    But we need to keep in mind that this is different from the “specification” concept that design advocates talk about.

    Given that I have never been able to make sense of the “specification” that ID proponents talk of, I can readily agree that they different.

  71. 71
    Eric Anderson says:

    groovamos @66:

    I don’t disagree with the idea that a mind must be involved with information. I have stated as much elsewhere and didn’t say anything to the contrary here.

    I was simply responding (a bit harshly, I realize, and I apologize) to your suggestion that I didn’t know what I was talking about because there were all kinds of things that could be ascertained by an observer about the string, and how it came about, and how it was transmitted and so on.

    It is not the case that I didn’t realize any of those things. I certainly realize them, have dealt with them in a prior post, and have moved on to a separate, more narrow issue in my exchange with Neil.

    Specifically, the whole point of my string example was to try and ascertain from Neil what he thinks the term “Shannon information” means, since he had made the (rather strange to me) statement that the string itself is the Shannon information.

    Please don’t read anything into my question to Neil beyond that. I generally agree with you that mind is quite relevant to the exercise of (i) producing information, and (ii) recognizing information. (Though this is not to ignore the very interesting question of whether information can exist objectively independent of an observer, which has been discussed on a prior thread).

    At any rate, I apologize if my response was too harsh. I’ve addressed the basic aspects of information in other threads and in this one I am trying to home in on a couple of rather technical points due to the confusion some have (as evidenced by some of the comments on this thread) about what the Shannon metric can and can’t tell us.

  72. 72
    Eric Anderson says:

    GBDixon @67:

    Thanks for the additional comments. Your example of the Rosetta Stone is precisely on point, and is an excellent example of information being there, available for the discovery. As is the information in the cell.

    I think I’m largely in agreement with your thoughts on that point. The only nuance, perhaps, is that I would suggest that we need to acknowledge that information can exist independently of the observer/recipient. Indeed, if we take the view that information only comes into existence at the point of recognition, then we have inadvertently collapsed two separate concepts into one: the existence of information and the recognition of that information.

    Anyway, I don’t mean to get into that here and go OT. We discussed it in some detail in a prior thread. I don’t think you had a chance to comment on the prior thread, but it might be interesting to read if you haven’t already, as it is very much on point with the issues you mentioned.

    http://www.uncommondescent.com.....ion-arise/

    Thanks again for the good thoughts.

  73. 73
    Eric Anderson says:

    Neil @70:

    Thanks. Let me see if I can phrase the point about specification another way.

    Take the string:

    TOBEORNOTTOBETHATISTHEQUESTION

    Now take the string:

    ETTTBSNHOBOORTUEHTISEEATQTNOIO

    Both of those strings have a “specification” as you are using the term, meaning that the string consists of a series of letters in a particular sequence. And if we were to write out that specification, we would simply be writing out the sequence. In other words, the way you are using the term “specification,” it simply means whatever sequence we happen to be looking at. And, yes, we can “specify” what sequence we are looking at by reciting the sequence. But that doesn’t add anything or tell us anything beyond the sequence itself.

    However, it is also obvious, I trust, that there is something substantively different between the two strings. And that substance doesn’t depend on the number of letters, or the number of particular kinds of letters, or even the number that will get spit out of a simple Shannon calculator.

    Rather, the difference lies in the fact that one string has underlying meaning, it has content, it specifies something (beyond the mere repetition of itself), it symbolizes something (again, beyond the mere repetition of itself). That is what we are talking about in the ID context by specification.

    Given that I have never been able to make sense of the “specification” that ID proponents talk of, I can readily agree that they different.

    It is rather basic, but let me see if I can take a stab at it. A specification in the context of a string as in our example, is the function, or the meaning, or the symbolic representation, as I’ve mentioned above.

    Again, take the two strings above. The second string can be described, it can be sequenced, it can be analyzed and calculated and poked and prodded and we can, through our faculties and tools, make lots of observations about that string.*

    We can also do the exact same thing with the first string and make every single similar kind of observation or analysis about it.

    And yet, when we are done with all that analysis, there is still something additional with the first string. Something symbolic, something meaningful, something representative — something that specifies something beyond itself. That is a specification for ID purposes.

    It is not mysterious or unusual or tricky. Indeed, in most cases it is so obvious that we tend to gloss over it or don’t give it a second thought. As we go about our lives we are literally swimming in specifications, so we take it for granted. So sometimes I find it is helpful to step back to a simple example just to home in on what we are talking about.

    Thanks,

    —–

    * Some people get confused and think that this means the string “contains” information. It doesn’t, as discussed in detail on the prior thread I referenced.

  74. 74
    Joe says:

    Neil Rickert:

    Given that I have never been able to make sense of the “specification” that ID proponents talk of, I can readily agree that they different.

    Well I, for one, disagree that they are different. For example, Stonehenge- we can assume there was a design/ engineering specification, ie something that said what to build, what type of stones and where to place those stones, even though no one has ever found such a thing. The pyramids, same thing. The Antikythera mechanism must have had a design specification but no one has found one.

    That said, IDists say we can identify a specification by the functionality and what it takes to get that functionality. If just about any configuration can produce the function then it isn’t that specified. If very few or only one configuration can produce it then we say it is specified.

    This all goes back to Crick’s notion of biological information and the functionally specific sequences it takes for living organisms to produce the proteins they need to survive.

  75. 75
    groovamos says:

    Eric: “I was simply responding (a bit harshly, I realize, and I apologize) to your suggestion that I didn’t know what I was talking about because there were all kinds of things that could be ascertained by an observer about the string, and how it came about, and how it was transmitted and so on.”

    I did not suggest such. Your example of the quality of the cargo placed on the truck suggested that information theory maybe has a shortcoming in some philosophical sense. And maybe the philosophical sense is the only way you can go at it because neither you or anyone else brings error-correction to the table, I did, and then GBD also correctly ran with that aspect. So I take your example and where the wrong items have been shipped I suggest what theory can only contribute in the scenario and that is provide a foundation to correct the errors, and then when someone doesn’t like the taste of what was shipped, their displeasure in text form can be transmitted error-free, or over a voice channel with lossy compression at minimum. I was hoping you appreciated attention to your blog by a couple of people with expertise in information theory. My suggestion that you maybe don’t know the mechanisms of error-free transmissions was based on the fact that the topic was not broached until my post, when errors were at the heart of your proposed scenario. And that you obviously were counting on error-free appearance on our screens of your text. I might put up another post about the topic because I have a thesis project going with information theory at the core of it.

  76. 76
    Eric Anderson says:

    Joe @74:

    “Well I, for one, disagree that they are different.”

    Joe, just to clarify, what Neil was referring to is the fact that in order to run a string (it could be any old random string) through a Shannon calculation, you have to “specify” what the string is. That is what he was calling a specification.

    As I mentioned to Neil, yes, it is true that you can “specify” a string by listing or reciting or entering into the system its sequence. It is true, but trivially true, and is not interesting for ID purposes. For ID, as you mentioned, we are interested in an underlying specification, a meaning, or function, or purpose, or representation.

    That is why his use of the word “specification” and what ID is talking about are not the same.

    If just about any configuration can produce the function then it isn’t that specified.

    I think that’s right.

    BTW, it is also part of the confusion Dembski ran into in his paper, Irreducible Complexity Revisited, which I addressed in my lengthy and now-largely-forgotten response paper.

  77. 77
    Joe says:

    Eric- Got it. However Neil originally said that specification, in the way it is used in the design industry, is a set of technical writings and/ or drawings/schematics pertaining to the object/ structure/ event. See Neil’s comment in #10.

    And that is why he doesn’t understand ID’s use. I was trying to explain that it is the same and we infer that specification from the calculated (biological) information/functional sequence specificity of whatever structure we are examining.

    That also means we are removed from the world of mere intelligent agency involvement to actual intentional design, which is what ID is really after, IMO.

  78. 78
    Eric Anderson says:

    Joe:

    Good catch on #10. I see what you are talking about.

    1. The “specification” Neil refers to in #10 is indeed a design-based specification, and is an example of what we are talking about with ID.

    2. However, the “specification” Neil refers to in #2 and #18 seems to be referring to the mere sequence of symbols that get plugged into a Shannon calculator or communication system.

    #1 is substantive, and is what we are interested in for ID purposes (as you note).

    #2 is merely descriptive, and is generally not what we are interested in for ID purposes.

  79. 79
    Eric Anderson says:

    groovamos @75:

    Thanks for the further comments. I’m not sure we have any real substantive disagreement, but perhaps I can clarify the purpose of my post, in case it wasn’t clear.

    I am aware of the issues regarding transmission and error correction. Had I wished to do a post on those topics or to give an example in that area, I would have. While those are important aspects of Shannon’s theory, to be sure, they are separate from the issue I am focusing on (and the one that so many anti-ID folks get hung up on), namely, the fact that the Shannon metric is not concerned with underlying meaning.

    My suggestion . . . was based on the fact that the topic [error-free transmission] was not broached until my post, when errors were at the heart of your proposed scenario.

    No. Errors in transmission were not at the heart of my analogy; they were irrelevant to it. My example assumes an error-free transmission.

    Again, the question of error-free transmission, correction, receipt, and even decoding, are not what I am focusing on. I am homing in on a very specific debating point that is all too often brought up by anti-ID proponents: namely, the claim that the mere generation of complexity (which, given relevant parameters, can be measured and assigned some numerical value using the Shannon metric) demonstrates that genetic algorithms, natural processes and the like can generate “information.”

    That is false. At least it is false in any meaningful sense and is only nominally true if we twist the definition of “information” to mean something without substance. Thus my post.

    As to the other issues that don’t directly relate to the debating point I am focusing on (but that I heartily agree are interesting in their own right — things like error-free transmission), I would certainly love to see your post if you get a chance, and I’m sure I and many others can learn much from your experience and expertise.

  80. 80
    Mung says:

    Well, Eric, in a sense all of this is about error correction!

    🙂

  81. 81
    Mung says:

    Eric:

    Whoa, slow down there cowboy! Let Neil get a word in edge-wise!

    LOL! ok, I just worked two days and two nights on minimal sleep. So Neil’s had his chance 🙂

  82. 82
    Mung says:

    Neil:

    Strictly speaking, it is Shannon information if it is part of a communication system or an information processing system.

    And this is the case because communications systems are intended to be used for communication and information processing systems are intended to process information.

    I sure hope someone is understanding the fundamental error of (some of) the critics. What does it mean to say that you have communicated a nonsensical string of characters?

    What happens to a communication overwhelmed by noise?

    What happens to an information processing system fed a sequence of nonsensical bits?

    Say you have a communication system and it is used to transmit a meaningless string of characters. Does it logically follow that communication can be meaningless?

    communication

    3 a : a process by which information is exchanged between individuals through a common system of symbols, signs, or behavior

    Neil:

    Strictly speaking, it is Shannon information if it is part of a communication system or an information processing system.

    Not really. *sigh*

    Neil:

    Marks on paper that just happen to look like letters would not count.

    What does this even mean? You mean meaningless marks?

    Wouldn’t the same hold true then for a sequence of meaningless symbols?

    Neil:

    This is why it is reasonable to say that Shannon information is always specified (by being entered into a communication system).

    Not really.

    Nothing *specifies* Shannon information. It can either be calculated or it cannot be calculated.

    What are the conditions that permit the calculation of Shannon information?

    You’ve certainly hinted at them in your post. Can you transition into “teaching mode” for a bit?

    Are there prerequisites for the calculation of the amount of Shannon information? What are they?

  83. 83
    Mung says:

    Eric:

    Shannon theory simply cannot say whether we are dealing with meaning or not.

    Close. So close. 🙂

    I agree.

    Assuming agreement that Shannon theory simply cannot say whether we are dealing with meaning or not, then how can Shannon theory say whether we are dealing with information or not?

    And if Shannon theory cannot tell us whether we are dealing with information or not, I say I have made my point. It is simply illegitimate to conclude from Shannon theory that information can be meaningless.

    Eric:

    So if information necessarily has meaning, as you appear to be arguing (and with which I would provisionally be inclined to agree), then — by definition — Shannon theory cannot tell us whether we are dealing with information or not. At least not in the underlying substance.

    I agree, though perhaps not for the same reason.

    Even if it were in fact the case that information can be meaningless, how does the conclusion that information can be meaningless in any way follow from Shannon theory?

    How does Shannon theory establish that we are in fact dealing with information?

    Even if information does not necessarily have meaning, Shannon theory simply cannot tell us whether or not information must have or does have meaning. That is simply not in the remit of the theory.

    It simply cannot tell us what information is.

    Eric:

    Thus the so-called “Shannon information” must be about something entirely different than the substance of the message, the substance of the string, the substance of the communication. The metric is measuring something separate from and apart from the underlying information.

    That is part of my whole point.

    I agree! Whee!

    It is “measuring” (calculating would be better) probabilities of symbols/messages given certain already understood information.

    Shannon Information is not itself meaningless. Why should it lead people to conclude that non-Shannon Information can be meaningless? Where’s the logic?

    I’ve argued all along that information is about something and that this is true as well of Shannon information.

    It’s commonly accepted that Shannon Information is information, but less well understood (by far) what it [Shannon Information] is information about.

    Questions?

  84. 84
    Mung says:

    Neil Rickert:

    This may be the most important point of the whole discussion. The word “information” is used in multiple conflicting ways. Unfortunately, people talk is if information were an objective or metaphysical entity. We would do better to understand it as an attribution appropriate for a particular use.

    That would also include “Shannon Information.”

  85. 85
    Mung says:

    GBDixonJuly 1, 2014 at 7:51 am

    Hi all,

    This will be my last comment. I have irritated our host and come off as a know-it-all. This is important to me, I guess.

    A shame. Even a know-it-all generally knows something!

    For what it is worth I had hoped to hear more from you.

    GBDixonJuly 1, 2014 at 7:51 am

    I think that a lot of confusion, argument and new vocabulary could be avoided if ID advocates knew information theory a little better.

    You see, I agree completely. But why would you say this then disappear? Don’t pontificate, educate!

    Many of us here at UD are utterly receptive to learning new things. This is especially true of information theory.

    Please stay.

  86. 86
    Neil Rickert says:

    Responding to Eric (#73):

    Take the string:

    TOBEORNOTTOBETHATISTHEQUESTION

    Now take the string:

    ETTTBSNHOBOORTUEHTISEEATQTNOIO

    Both of those strings have a “specification” as you are using the term, meaning that the string consists of a series of letters in a particular sequence.

    Right. Shannon’s theory is a theory of communication, so specification ought to mean the specification of what is to be communicated.

    However, it is also obvious, I trust, that there is something substantively different between the two strings.

    No, not at all. There’s a difference in how we react to them. But that’s a difference in our reaction, not a substantive difference in the strings.

    Rather, the difference lies in the fact that one string has underlying meaning, it has content, it specifies something (beyond the mere repetition of itself), it symbolizes something (again, beyond the mere repetition of itself).

    No, that’s nonsense. Neither string has any meaning. We contribute the meaning. Meaning is not a property of a string. Rather, meaning comes from us. Meaning is inherently subjective. But none of that subjective quality that we call “meaning” is ever present in the communication channel.

    If you want an objective science of communication or of information, then you must leave meaning out of that science.

  87. 87
    Mung says:

    Neil Rickert:

    If you want an objective science of communication or of information, then you must leave meaning out of that science.

    Bullshit!. If Shannon Information isn’t meaningful, then of what use is it?

  88. 88
    Eric Anderson says:

    Mung @83:

    I’ve argued all along that information is about something and that this is true as well of Shannon information.

    It’s commonly accepted that Shannon Information is information, but less well understood (by far) what it [Shannon Information] is information about.

    There is no question that the Shannon measurement/calculation is about something.

    The big problem comes because people think they are calculating the amount of “information” in a string when they run the Shannon calculation. They aren’t. Indeed, the only thing they are doing is plugging some parameters into a formula and producing a measurement or a calculation result. It is very unfortunate that this measurement has come to be called (by some people) “Shannon information” as many people then trick themselves into thinking that the measurement has somehow quantified the amount of “information” in the string.

    We might as well take the temperature of an object in degrees Kelvin and call the result “Kelvin information.” Or we could plug some numbers into Einstein’s most famous equation and pronounce the result “Einstein information.”

    Of course we will have produced information when take a measurement and assign it a value according to an agreed-upon system, or when we plug parameters into an equation and produce a result. Yes, that result is “information.” Yes, as a result of our mental faculties, in concert with accepted linguistic and mathematical conventions, we produce new information when we express the results of a measurement or when we set forth the results of a calculation. Those resulting numbers are information in their own right. But it doesn’t mean those numbers represent a measurement of information.

    The resulting numbers themselves could be plugged into a Shannon calculator to yield a number. And that number is of course itself another piece of information. Which can then be fed back into the Shannon calculator itself — just like any other piece of information — to yield another number. And the cycle can continue . . .

    The problem — what is really a very simple conceptual problem, but which has caused nearly endless difficulties for some people — is that (unlike in nearly every other calculation that we run in any other field), some people insist on calling the result of the Shannon calculation “Shannon information.” Couple that with the fact that information is generally plugged into the Shannon calculation on the front end, and many people get quite confused into thinking that the “Shannon information” coming out the back end somehow represents a measurement of the amount of “information” going in the front end.

    It is easier to keep things separate in a case like Einstein’s equation, for example. We can see that if we take mass and multiply it times the speed of light squared then we will get a number that represents energy.

    Unfortunately, in the Shannon case, people see “information” plugged into one end and “Shannon information” coming out the other end and think that this little number must somehow represent the amount of “information” that was contained in the input string.

    At any rate, the upshot of all this is that the term “Shannon information” is a tremendous stumbling block, particularly for individuals who are approaching the topic (hoping to discredit the design inference) by claiming variously that (i) everything contains information, or (ii) information can be meaningless, or (iii) the Shannon number measures the actual information content, or (iv) some combination of the foregoing absurdities.

    Yes, those of us who are a bit more careful can keep in mind what “Shannon information” really means and what it is really about and can avoid getting confused. But it is a very tough row to hoe to try to explain the issues to someone who is already entrenched in misunderstandings and who, unfortunately in so many cases, has a mental or philosophical predisposition to not understand the issues.

  89. 89
    Eric Anderson says:

    Neil @86:

    Shannon’s theory is a theory of communication, so specification ought to mean the specification of what is to be communicated.

    Again, all your “specification” means is the sequence of the string that is plugged into the Shannon calculation. Yes, we can call that a “specification,” but it doesn’t have anything to do with a substantive concept of specification, which is what ID is about.

    Neither string has any meaning. We contribute the meaning. Meaning is not a property of a string. Rather, meaning comes from us.

    Of course an intelligent being was involved in assigning meaning to a string. And because there is a symbolic connection between the real-world meaning and the string, we recognize that the string contains representation, meaning, substance. And we also know, just speaking of ID for a moment, that such a situation is only known to arise by the activity of intelligent beings.

    If you’re arguing that meaningful information/communication doesn’t arise without the activity of an intelligent being to provide that meaning, then I agree. Welcome to the ID side! 🙂

    If you want an objective science of communication or of information, then you must leave meaning out of that science.

    Look, Shannon was interested in quantifying bits for transmission and communication purposes. And his theory is not interested in the meaning of the underlying communication. We seem to agree on that point.

    With that in mind, can you nevertheless appreciate that meaning of a communication can be of importance, in its own right and irrespective of whatever optimal number of bits gets spit out of a Shannon calculation? Can you accept that ID is interested primarily in the former?

    —–

    The upshot of all this is that the primary point of my post is underscored: generating random strings that have a high Shannon calculation result do not — and by definition, cannot — invalidate the design inference, because the design inference is not based on simply having a high Shannon calculation number.

  90. 90
    Neil Rickert says:

    Eric (#89):

    And because there is a symbolic connection between the real-world meaning and the string, we recognize that the string contains representation, meaning, substance.

    No, there is no such symbolic connection, as far as I can tell. The string is symbolic, but not the connection.

    The only people that I know who would claim (without proof) that the connection is symbolic, are those AI proponents at the extremes of materialism, who claim that humans are just symbol processing computers.

    With that in mind, can you nevertheless appreciate that meaning of a communication can be of importance, in its own right and irrespective of whatever optimal number of bits gets spit out of a Shannon calculation? Can you accept that ID is interested primarily in the former?

    In effect, that amounts to saying that ID is a quest to find an objective account of the subjective. That, too, puts you in agreement with those at the extremes of materialism.

    Personally, I see any such quest as doomed to fail.

  91. 91
    Eric Anderson says:

    Neil:

    You are confusing two different things. Just because something is created by an intelligence doesn’t mean it is subjective. If I discover a code and can determine what the symbols refer to, then I have learned something real about the code — objectively, just like with mathematics or the genetic code. There is nothing “subjective” about it. If I discover an artifact or a machine and by reverse engineering am able to ascertain its function, there is nothing subjective about that function — it exists, it is real. Don’t make the category mistake of lumping everything that is produced by a mind or by an intelligence into the category of “subjective” (which can then be blithely dismissed without applying intellectual effort to the question at hand).

    Furthermore, the attempt to paint the design inference as dealing with things that are “subjective” is no more becoming than the attempt to paint it as dealing with things that are “supernatural.” Both claims are but a rhetorical attempt to dismiss the substance — you know, if it is “supernatural” it isn’t “science”; if it is “subjective” it isn’t objective, and therefore not worth considering.

    Not an intellectually worthy approach to take.

  92. 92
    Mung says:

    Hi Eric,

    Sorry, I’ve been occupied with so many other things lately. I haven’t had the time to spend on this that I would like. Perhaps later this week and into next will improve.

    But I wanted to get back to you on something.

    Eric:

    Shannon information = string of characters

    Mung @ 47:

    This is what Elizabeth Liddle thought. Toss a coin x number of times you get y bits of “Shannon information.”

    The coin tosses were meaningless. When asked what they were about she had no answer. But because y bits of Shannon Information could be calculated it somehow meant that information could be meaningless.

    Unfortunately for ID, that is also the view presented by Stephen Meyer in Signature in the Cell.

    Eric:

    Do you have a cite for this? I could maybe look it up, but I’m too lazy and I might not find the actual passage that you are referring to, so if you have a quote handy, that would be helpful.

    For example:

    He [Jones] knows that all long-distance phone numbers have ten characters and are constructed from the same ten digits. He writes down the digits 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, on another piece of paper. He gives a sigh of relief. Now he has the raw materials, the building blocks, to generate Smith’s ten digit phone number.

    He quickly arranges the digits on his list at random into a sequence of ten characters. Now he has some information, some Shannon information. Being mathematically inclined, he quickly calculates how much information he has using Shannon’s familiar formula. He’s relieved and impressed. His new ten-digit phone number has a lot of information – 33.2 bits to be exact.

    Signature in the Cell p. 105

    I can come up what more, and will, time permitting.

    Meyer is simply confused here (and confusing). By simply arranging ten characters into a string of characters “at random” one does not create information, not even Shannon Information.

    And his new ten-digit phone number may not even be a phone number! As Meyer admits.

  93. 93
    Mung says:

    Neil @ 2:

    Just to be clear, the metric is not the information.

    You’re simply mistaken. The metric is the information. Else it would not be informative, it would not be a metric, it would not be objective, it would be meaningless.

    The metric is a way of measuring the amount of information (or the channel capacity).

    You’re simply mistaken. The metric does not “measure the amount of information.” There is no measure for information.

  94. 94
    Neil Rickert says:

    Eric (#91):

    You are confusing two different things. Just because something is created by an intelligence doesn’t mean it is subjective.

    You seem to have missed the point.

    I’m saying that meaning is inherently subjective. Whether or not meaning is created by an intelligence, or is just a natural part of organic life, is not relevant to the point that meaning itself is subjective.

    If you insist on tying information to meaning, then that makes information subjective. Again, whether or not the information is created by an intelligence is not relevant. It’s the fact that information, as you use that term, is tied to meaning, together with the fact that meaning is subjective.

  95. 95
    Eric Anderson says:

    Mung, thanks for taking time for a few additional comments.

    I hope you aren’t quoting me for the notion that:

    Shannon information = string of characters.

    That was my assessment of Neil’s confused approach @31.

    —–

    Meyer is simply confused here (and confusing). By simply arranging ten characters into a string of characters “at random” one does not create information, not even Shannon Information.

    I think you are taking your criticism a bridge too far. If we are saying that “Shannon information” is the resulting number that is spit out when we plug a string into a Shannon calculation, then, yes, Meyer should have been more careful in his light-hearted analogy.

    But this brings us back to the very nuanced point: Does a string “contain” Shannon information?

    If we are defining Shannon information as the mere numerical result spit out of a calculation, then perhaps no. If we are defining the result spit out of a calculation as a measure of something (say, information carrying capacity), then perhaps yes.

    And his new ten-digit phone number may not even be a phone number! As Meyer admits.

    Meyer didn’t “admit” anything on that front. It was part of his point. So you should have said, “As Meyer underscores.”

    —–

    We seem to be circling around a definitive definition, and I suspect we are very much in agreement.

    However, what I would love to hear from you (notwithstanding your wonderful and energetic Socratic method of asking lots of questions), is your definition of Shannon information.

    Let’s say I approach you on the street and ask you for a straight-up, unambiguous, no-hidden-nuances, non-rhetorical definition of “Shannon information,” what would you say?

  96. 96
    Mung says:

    Neil Rickert:

    I’m saying that meaning is inherently subjective.

    So you’re NOT saying that Shannon Information is meaningless, just that it’s subjective. Thanks for clearing that up. Whew!

    Neil Rickert:
    If you insist on tying information to meaning, then that makes information subjective.

    Make up your mind, please.

    Information is inherently tied to meaning. There is no such thing as meaningless information. The very idea is absurd.

    Are you asserting that Shannon Information is not tied to meaning? If it isn’t, then what use is it?

    Or are you saying Shannon Information is subjective? If so, so what?

  97. 97
    Mung says:

    Eric:

    Meyer didn’t “admit” anything on that front. It was part of his point. So you should have said, “As Meyer underscores.”

    What was Meyer’s point that he underscores by admitting that the ten digits may not even constitute a phone number? Was it that a random string of characters can provide meaningless information?

  98. 98
    Eric Anderson says:

    Mung, Meyer’s point by underscoring that ten digits may not even constitute a valid phone number is pretty clear. The whole point of those couple of pages is to get people thinking about the difference between mere complexity and specified complexity. His simple example is perfectly suited to that task.

    . . . Smith’s sequence exhibits what has been called specified complexity, while Jones’s exhibits mere complexity.

    p.107

    This is true regardless of whether we define Shannon information as being a property of the string itself or the resulting measurement number of the Shannon calculation.

    Again, I’m not sure why you are hung up on this.

    More interesting to me though, is how you would define “Shannon information.”

  99. 99
    Mung says:

    Meyer provides a good starting point for discussing Shannon Information.

    Shannon’s theory of information was based upon a fundamental intuition: information and uncertainty are inversely related.

    Signature in the Cell p. 88

    Uncertainty, of course, implies uncertainty about something. So Shannon Information in no way reduces the requirement for the aboutness of information. In fact, it is inherent in the very concept of Shannon Information.

    Claude Shannon wanted to develop a theory that could quantify the amount of information stored in or conveyed across a communication channel. He did this first by linking the concepts of information and uncertainty and then by linking these concepts to measures of probability.

    According to Shannon, the amount of information conveyed (and the amount of uncertainty reduced) in a series of symbols or characters is inversely proportional to the probability of a particular event, symbol, or character occurring.

    Signature in the Cell p. 88

    The amount of uncertainty about what?

    One wonders why Shannon Uncertainty isn’t as popular a concept as Shannon Information. 🙂

    By equating information with the reduction of uncertainty, Shannon’s theory implied a mathematical relationship between information and probability. Specifically, it showed that the amount of information conveyed by an event is inversely proportional to the probability of it’s occurrence. The greater the number of possibilities, the greater the improbability of any one being actualized, and thus the more information that is transmitted when a particular possibility occurs.

    Signature in the Cell p. 89

    To calculate the amount of Shannon Information one must be able to specify the amount of Shannon Uncertainty. And when that is done, it becomes readily apparent what Shannon Information is about. (And what it is not about.

  100. 100
    Mung says:

    But this brings us back to the very nuanced point: Does a string “contain” Shannon information?

    It is a very nuanced point and many people seem to be unaware of the nuance or even of the possibility of the necessity for any nuanced view. The answer is no.

    Strings of characters do not “contain” or “carry” or “convey” Shannon Information. Think of Shannon Information as meta-information.

    Shannon information = string of characters

    Sorry, I just needed someplace to start and that seemed to be convenient at the time. Didn’t mean to imply that was your PoV. 🙂

  101. 101
    Mung says:

    Consider two sequences of characters:

    “Four score and seven years ago”

    “nenen ytawoi jll sn mekhdx nnx”

    Both of these sequences have an equal number of characters. Since both are composed of the same 26-letter English alphabet, the amount of uncertainty eliminated by each letter (or space) is identical. The probability of producing each of those two sequences at random is identical. Therefore, both sequences have an equal amount of Shannon information as measured by Shannon’s theory.

    Signature in the Cell p. 90

    Meyer is mistaken, but how (or in how many ways) and why?

  102. 102
    Eric Anderson says:

    Mung, you are a genius at writing riddles. Less so at clear exposition of an answer.

    What is Shannon information? Please define.

  103. 103
    Mung says:

    Eric, see my response at 99.

    You don’t agree with Meyer’s definition of Shannon Information or you don’t think he is explaining what Shannon Information is in the quoted material?

  104. 104
    Mung says:

    RE: 101

    How does Meyer know that these characters are from the English alphabet and not from the Latin alphabet?

    Since both are composed of the same 26-letter English alphabet, the amount of uncertainty eliminated by each letter (or space) is identical.

    The fact that symbols may appear to be from the same alphabet doesn’t determine the frequency or probability of their occurrence in a sequence of characters or symbols, and that is what determines how “informative” a given symbol is.

    The probability of producing each of those two sequences at random is identical.

    But letters in the English language are not equally probable, so different letters convey different amounts of Shannon Information.

    Therefore, both sequences have an equal amount of Shannon information as measured by Shannon’s theory.

    The conclusion does not follow. It’s a non sequitur.

    There’s no reason to believe both sequences were produced at random. In fact, I think we can be pretty sure the first one was not produced at random.

    Perhaps the second sequence came from a source which uses an alphabet identical to that of the first sequence (say one containing 26 symbols plus a space), and perhaps each symbol was equally likely to be transmitted by the source with the same probability (though given the number of spaces I have my doubts), and each character was selected randomly and transmitted.

    What the average info per symbol according to Shannon’s theory?

    But I find it extremely difficult to believe that the first sequence came from the same source. And why should I?

    And if it did not come from a source with the same characteristics, then it doesn’t have “an equal amount of Shannon information as measured by Shannon’s theory.”

  105. 105
    Mung says:

    In terms of the colloquial meaning of information, Shannon’s paper deals with the carriers of information – symbols – and not with information itself. It deals with communication and the means of communication rather than that elusive end product of communication – information.

    Information Theory and Coding, p. 2

    We shall learn that the symbols must obey certain laws if they are to be capable of transmitting information

    Information Theory and Coding, p. 2

  106. 106
    Mung says:

    : Information: The New Language of Science
    : Hans Christian von Baeyer
    : Chapter 4
    : Counting Bits: The scientific measure of information
    : p. 28

    In contrast to the vague verbal definition of information, the technical definition, though skeletal, is a model of specificity and succinctness. Claude Shannon, the founder of information theory, invented a way to measure ‘the amount of information’ in a message without defining the word ‘information’ itself, nor even addressing the question of the meaning of the message. He produced, in effect, an operational definition like that of temperature, except that his measuring device – a simple recipe – is not a physical apparatus, like a thermometer, but a conceptual tool.

    Shannon’s information measure is most easily applied to a message that consists of a string of binary possibilities – yes or no, heads or tails, zero or one – each of which is equally likely at every step along the string. According to Shannon, each choice corresponds to one bit (short for ‘binary digit’) of information. Communicating to a friend the outcome of three consecutive tosses of a penny, for example, requires three bits of information, which would define eight possible strings of heads and tails. More generally, Shannon’s recipe is simple: To find the information content of any message, translate the message into the binary code of the computer and count the digits of the resulting string of zeros and ones. The number so obtained is called ‘Shannon information’, and the technique is known, somewhat dismissively, as bit-counting.

  107. 107
    Mung says:

    : Information: The New Language of Science
    : Hans christian von Baeyer
    : Chapter 12
    : Randomness: The flip side of information
    : p. 99

    Shannon’s technical definition of the information content of a message – the number of digits when the message is written in the binary code of the computer – doesn’t distinguish between sense and nonsense.

  108. 108
    Mung says:

    Eric, do we have a definition yet of Shannon Information?

    If not, why not?

  109. 109
    Mung says:

    Serious difficulties arise when scientists try to separate the idea of probability from the idea of information, because the first cannot be defined without the help of the second. In Shannon’s theory, entropy is a probability distribution, assigning various probabilities to a set of possible messages. But entropy is also a measure of what the person receiving a message does not know about it before it arrives. Entropy is an index of his uncertainty as to what to expect.

    – Grammatical Man: Information, Entropy, Language and Life, p. 92-93

  110. 110
    Mung says:

    More on the definition of Shannon Information.

    Formal-mathematical information A third class of information-theoretic notions includes the formal concepts of information, which have been initially introduced as mathematical tools for measuring the performance of communications devices. The classical notion, in this category, was introduced by the mathematical theory of communication of Shannon (1948) and Shannon and Weaver (1949). In the latter, [Shannon] information is a measure of one’s freedom of choice when one selects a message (the logarithm of the number of available choices or of probabilities).

    – Nicolas J. Bullot. Attention, Inforamtion, and Epistemic Perception. In Inforamtion and Living Systems: Philosophical and Scientific Perspectives. George Terzis and Robert Arp, eds.

  111. 111
    Mung says:

    P1 The Elementary Problem: What Is Information?

    Information can be viewed from three perspectives… Many extensionalist approaches to the definition of information as reality or about reality provide different starting points for answering P1.

    1. the information theory approach (mathematical theory of codification and communication of data/signals, Shannon and Weaver (1949 rep. 1988) defines [Shannon] information in terms of probability space distribution;

    Each extensionalist approach can be given an intentionalist reading, by interpreting the relevant space as a doxastic space, in which information is seen as a reduction in the degree of uncertainty or level of surprise in an informee, given the state of information in the informee.

    Information theory in (1) approaches information as a physical phenomenon, syntactically. It is not interested in the usefulness, relevance, meaning, interpretation, or aboutness of data, but in the level of detail and frequency in the uninterpreted data (signals or messages). It provides a successful mathematical theory because its central problem is whether and how much data, not what information is conveyed.

    – Luciano Floridi, The Philosophy of Information. p. 30-31

  112. 112
    Mung says:

    Hey Eric,

    In case you ever find time to get back to this thread.

    I was reading a book recently that indicated that there are four different Shannon entropies. Would it follow that there are four different measures and that there are thus perhaps four different definitions of “Shannon Information”?

  113. 113
    Mung says:

    Let’s identify another mistaken ID argument about Shannon Information.

    Shannon’s classical information theory concerns itself only with statistical relationships of material symbols found within the code of Universal Information. This was because nothing more was necessary in order to address the technical issues of information transmission and storage. While Shannon stated this point clearly in his landmark paper, most modern day evolutionary theorists champion his definition primarily because it allows for the creation of ‘information’ by randomly assembling symbols. This makes creation of biological information trivial, and separate biological information from biological functionality. The attempt to define biological information in this way is clearly ideologically driven and is obviously not sufficient, since no thinking person would exclude meaning and purpose from biological (functional) information.

    – Biological Information: New Perspectives. p. 16

    There are many things I find wrong here, but I’ll focus on just one.

    Notice the clear connection to the same mistake that Meyer makes.

    Shannon’s theory does not and cannot tell us what is required for the creation of information. It does not and cannot ‘allow’ for the creation of ‘information’ by randomly assembling symbols.

  114. 114
    Mung says:

    Shannon Information is still information. It’s not as if with the appearance of Shannon’s Theorems information gained a little brother, a distinct and separate entity.

    However, it is confined to something specific within the greater realm of information: probability distributions.

  115. 115
    Mung says:

    Shannon information is not really information.

    Shannon Information really is information.

    …a large part of the problem results from the use of the word “information” in reference to the Shannon metric. As I have stated before, somewhat provocatively, we would all have been better off if instead of “Shannon information” the concept were referred to as the “Shannon measurement” or the “Shannon metric.”

    So let’s call it Shannon’s Measure of Information (SMI).

    Eric Anderson @ 102:

    What is Shannon information? Please define.

    Your OP is titled “Intelligent Design Basics – Information – Part III – Shannon”

    Did you fail to define Shannon Information in the OP? Have you defined Shannon Information anywhere in this thread?

    Perhaps we can work together to define Shannon Information.

    Assuming, of course, that you are willing to dispense with the claim that Shannon Information is not really information.

    A fundamental, but somehow forgotten fact, is that information is always information about something.

    – Jan Kahre, The Mathematical Theory of Information

    What is Shannon’s Measure of Information (SMI) about?

    To assert that Shannon information is “not really information” is to assert that there is no “aboutness” to Shannon Information. This assertion ought to be discarded as absurd on it’s face.

  116. 116
    Mung says:

    “The meaning of information is given by the processes that interpret it.”

    – Ed Fredkin

  117. 117
    Mung says:

    “He [MacKay] proposed that both Shannon and Bavelas were concerned with what he called ‘selective-information,’ that is information calculated by considering the selection of message elements from a set.”

    http://www.physics.utoronto.ca.....h2wii2.pdf

  118. 118
    Mung says:

    Shannon Information: A Misleading Analogy

    From the OP:

    My only task, the only thing I have ever claimed to be able to do, is calculate the maximum carrying capacity of the truck with the given boxes.

    Shannon’s Capacity Theorem

    See also:

    The Shannon limit or Shannon capacity of a communications channel is the theoretical maximum information transfer rate of the channel, for a particular noise level.

    Noisy-channel coding theorem

    In information theory, the Shannon–Hartley theorem tells the maximum rate at which information can be transmitted over a communications channel of a specified bandwidth in the presence of noise. It is an application of the noisy channel coding theorem to the archetypal case of a continuous-time analog communications channel subject to Gaussian noise. The theorem establishes Shannon’s channel capacity for such a communication link, a bound on the maximum amount of error-free digital data (that is, information) that can be transmitted with a specified bandwidth in the presence of the noise interference, assuming that the signal power is bounded, and that the Gaussian noise process is characterized by a known power or power spectral density. The law is named after Claude Shannon and Ralph Hartley.

    Shannon–Hartley theorem

    In electrical engineering, computer science and information theory, channel capacity is the tightest upper bound on the rate of information that can be reliably transmitted over a communications channel. By the noisy-channel coding theorem, the channel capacity of a given channel is the limiting information rate (in units of information per unit time) that can be achieved with arbitrarily small error probability.

    Information theory, developed by Claude E. Shannon during World War II, defines the notion of channel capacity and provides a mathematical model by which one can compute it. The key result states that the capacity of the channel, as defined above, is given by the maximum of the mutual information between the input and output of the channel, where the maximization is with respect to the input distribution

    Channel capacity

    There’s more to Shannon’s paper than just the channel capacity, and it’s this “something more” that people have in mind when they think of Shannon Information.

    It’s in Section 6 (Choice, Uncertainty and Entropy) starting on page 392.

    So to expand the analogy, Shannon didn’t just compute the carrying capacity of the truck, he developed a way to quantify (measure) the “sizes” (or average size) of the boxes.

Leave a Reply