In this post I want to consider another aspect of information. Specifically, I want to consider the concept of “Shannon information.”
First of all, I admit to having ruffled a few feathers when I mentioned in passing in a prior post that “Shannon information is not really information.” As I have also written before in comments on UD, I don’t begrudge anyone referring to the Shannon metric as “information.” That terminology has penetrated the English language and has become regularly-used in information theory. So, no, I am not going to police everyone who puts the words “Shannon” and “information” next to each other.
However, no small amount of misunderstanding has resulted from the unfortunate term “Shannon information.” In particular, as it relates to intelligent design, some critics have seized on the idea of Shannon information and have argued that because this or that computer program or this or that natural process can produce a complex string or a complex sequence, that therefore such a program or process is producing new complex “information.” This proves, the argument goes, that purely natural processes can produce new and large amounts of information, contra the claims of intelligent design.
Such thinking demonstrates a lack of understanding of CSI – in particular the need for specification. However, a large part of the problem results from the use of the word “information” in reference to the Shannon metric. As I have stated before, somewhat provocatively, we would all have been better off if instead of “Shannon information” the concept were referred to as the “Shannon measurement” or the “Shannon metric.”
Claude Shannon published a paper entitled “A Mathematical Theory of Communication” in the July 1948 volume of The Bell System Technical Journal. This paper is available online here and is considered a foundational groundwork for not only Shannon’s subsequent research on the topic, but for information theory generally. To be sure, there are many other aspects of information theory and many other individuals worthy of acclaim in the field, but Shannon is perhaps justifiably referred to as the father of information theory.
But before delving into other details in subsequent posts, time permitting, I want to relate a short experience and then a parable. Consider this a primer, a teaser, if you will.
The Warehouse
When I was a teenager in high school, one of my part time jobs was working in a warehouse that housed and sold equipment and materials for the construction industry. On a regular weekly schedule we would load a truck with supplies at the main warehouse and drive the truck to a smaller warehouse in a different city to supply the needs in that locale. The day of the week was fixed (if memory serves, it was generally a Friday) and the sending warehouse foreman made sure that there were enough people on hand in the morning to pull inventory and load the truck, while the receiving warehouse foreman in turn ensured that there were enough people on hand in the afternoon to unload the truck and stock the inventory.
Due to the inevitable uneven customer demand in the receiving city, the needs of the receiving warehouse would vary. With good inventory management, a large portion of the receiving warehouse’s needs could be anticipated up front. However, it was not uncommon for the receiving warehouse to have a special order at the last minute that would necessitate removing a large crate or some boxes from the truck that had already been loaded in order to make room for the special order. At other times when no large orders had been made, we would finish loading all the supplies and find that we still had room on the truck. In this latter case, the sending foreman would often decide to send some additional supplies – usually a high turnover item that he knew the receiving warehouse would likely need shortly anyway.
In either case, the goal was to make most efficient use of the time, money and expense of the truck and driver that were already slated to head to the other town – taking the best possible advantage of the previously-allocated sunk costs, if you will. Ensuring that the shipment container (in this case a truck) made best use of the available capacity was a key to efficient operations.
I want to now take this experience and turn it into a parable that relates to Shannon information.
The Parable of the Fruit Truck
Let’s assume that instead of heating and cooling equipment and supplies, the warehouse sells fruit directly to customers. Let’s further assume that the various kinds of fruit are shipped in different-sized boxes – the watermelons in one size of box, the pineapples in another, the apples in another, and the strawberries in yet another.
Now, for simplicity, let’s suppose that customers purchase the fruit on a long-term contract with a pre-set price, so the primary variable expense of the warehouse is the expense of operating the truck. The warehouse would thus be highly incentivized to maximize the efficiency of the truck – sending it out on the road only as often as needed, and maximizing the carrying capacity of the truck.
The dock workers in our parable, however, are not particularly sharp. As the fruit comes in from the farms, the dock workers, without confirming the contents, simply start packing the boxes at the front of the truck, working their way to the back. Invariably, there are gaps and open spaces as the various-sized boxes do not precisely conform to the internal capacity of the truck. Some days are better than others by dint of luck, but the owner quickly realizes that the packing of the truck is inefficient. Worse still, customers regularly complain that (i) the truck is arriving only partly filled, (ii) boxes contain the wrong kind of fruit, or (iii) in particularly egregious cases, the boxes contain rotten fruit or no fruit at all.
As a result, the warehouse owner decides to hire a sharp young man fresh from the university whose sole job it is to figure out the best way to pack the truck, to create the most efficient and time-saving way to deliver as much fruit as possible given the carrying capacity of the truck.
Let’s say this young man’s name is, oh, I don’t know, perhaps “Shannon.”
Now our hero of the parable, Shannon, works in the office, not the loading dock, and is unable to confirm the actual contents of the boxes that are loaded on the truck. Further, he quite reasonably assumes the dock workers should be doing that part of the job. Notwithstanding those limitations, Shannon is a sharp fellow and quickly comes up with a formula that gives the owner a precise calculation of the truck’s carrying capacity and the exact number of each type of fruit box that can be loaded on the truck to ensure that every square inch of the truck is filled.
Elated with the prospect of putting all the customer complaints behind him, the warehouse owner hands down the instruction to the dock workers: henceforth the truck will be packed with so many watermelon boxes, so many pineapple boxes, so many apple boxes and so on. Furthermore, they will be packed according to Shannon’s carefully worked out order and placement of the boxes.
After the next week’s shipments, the owner is surprised to receive a number of customer complaints. Although not a single customer complains that the truck was only partly full (it was packed tightly to the brim in all cases), several customers still complain that (i) boxes contain the wrong kind of fruit, or (ii) in particularly egregious cases, the boxes contain rotten fruit or no fruit at all.
Furious, the owner marches to Shannon’s desk and threatens to fire him on the spot. “I hired you to figure out the best way to pack the truck to create the most efficient approach to delivering as much fruit as possible! But I am still swamped by customer complaints,” he fumes as he throws down the list of customer complaints on Shannon’s desk. Unfazed, Shannon calmly looks at the customer complaints and says, “I understand you used to get complaints that the truck was only partially filled, but I notice that not a single customer has complained about that problem this week. You hired me to find the most efficient delivery method, to ensure that the truck was maximizing its carrying capacity of boxes. I did that. And that is all I have ever claimed to be able to do.”
“But some of the customers got the wrong fruit or got no fruit at all,” sputters the owner. Based on your work we told them they would be receiving a specific quantity of specific types of fruit each week.
“I’m sorry to hear that,” retorts Shannon, “but you should not have promised any specific fruit or any particular quantity of fruit based on my formula alone. From my desk I have no way of knowing what is actually in the boxes. The supplier farms and dock workers can answer for that. What is in the boxes – what is actually delivered to the customer – has nothing to do with me. I have no ability from where I am sitting, nor frankly any interest, in guaranteeing the contents of the boxes. My only task, the only thing I have ever claimed to be able to do, is calculate the maximum carrying capacity of the truck with the given boxes.”
The Analogy
The fruit truck is obviously but a simple and fun analogy. However, it does, I believe, help newcomers get a feel for what Shannon can do (analyze maximum carrying capacity of a delivery channel) and what Shannon cannot do (analyze, confirm, understand or quantify the underlying substance). We’ll get into more details later, but let’s kick it off with this analogy.
What similarities and differences are there between our parable of the fruit truck and Shannon information? What other analogies are you familiar with or perhaps have yourself used to help bring these rather intangible concepts down to earth in a concrete way for people to understand?
Nota bene:
I drafted this post more than two months ago, meaning to work on it more before publishing. Unfortunately, work obligations and life generally have prevented me from spending further time on it. As a result, I am publishing it “as is” in the hope it might nevertheless serve as a point for discussion.
Just to be clear, the metric is not the information. The metric is a way of measuring the amount of information (or the channel capacity).
I am inclined to say that all Shannon information is specified. Noise signals are not usually considered to be Shannon information.
Your parable is a poor illustration. Shannon was concerned with correct transmission of information over a noisy channel. Your parable ignores the concern with correctness. It is perhaps an analogy for a noisy channel, but it is misleading as an account of Shannon information.
Neil, who could know where to start with your comments in #2.
Listen to what Claude Shannon is telling you:
He is telling you that there is a message to be communicated from one point to another point, where it will be selected at that point. It will be selected — because it is specified, Neil. Representations have to be related to their meaning at the receiver in order for the message to be communicated. It’s the distinction between me writing a meaningless scribble on a piece of paper, or, writing the word “apple”. You recognize the “apple” because it’s specified; you know what those particular scribbles mean. Shannon is telling you that this process of selection at the receiver is irrelevant to the problem of engineering a communication channel that can carry any scribble – the one selected and all the others that are not, i.e. the noise. That is what the system has to be able to do in order to function.
You need a complete overhaul.
Ahh. Forgot the two words that clarify the point:
He is telling you that there is a message to be communicated from one point to another point, where it will be selected at that point [as well]. It will be selected — because it is specified
EA et al, yes, Shannon Info is really an info carrying capacity metric. In particular the entropy in a message is the average info capacity per symbol, which is linked to the info approach to entropy. (Entropy, on such a view, is a metric of average missing info to specify microstate given only a macro-level description of a thermodynamic system; which forces us to treat the microstate as effectively rndom, in trying to extract work, etc.) That is why functionally specific complex organisation and associated info is a needed further step. KF
UB: Yes, the detection of a communicated message in the face of the interference of noise is a case of inference to design, to intelligent message rather than noise. We need to ponder the significance of the metric signal to noise power ratio that implies that there are characteristic . . . empirically characteristic . . . differences between overwhelmingly typical messages and overwhelmingly typical noise. This also appears in filter theory where noise as a rule is high frequency “grass” on a CRO screen — in the case of the classic Telequipment D52 (I think eventually bought out by Tektronix), that was very literal as the screen is bright green. Signals tend to be much narrower band, centred on a carrier wave. And yes I am familiar with frequency hopping and other spread spectrum techniques. They boil down to imposing a code that if known allows us to pull a signal almost like magic out of what would otherwise appear to be noise. Just think of the old electronics rule of thumb to avoid differentiators like the plague as this is a noise amplifying process. KF
The comments are correct: The set of valid messages among the total of all possible messages that can be received defines the information. Channel capacity has no meaning without this message list and information entropy and the channel capacity cannot be calculated without knowing the number of valid messages among the number of possible messages.
When both a sender and receiver are involved, they decide together the set, or list, of valid messages: THEY decide what the information is. In SETI and molecular biology, there may be senders but their decisions regarding messages must be inferred as we cannot establish an a priori message list. In SETI we look for message sequences that are unlikely to be caused by natural events.
In biology there are DNA sequences that are clearly information because they code for functional proteins. Some other sequences may not be clearly information, and we can argue whether they are or not, so the exact information content of a gene may be unknown but it has a lower bound. The set of all possible DNA sequences constitute the possible messages that can be received (by the molecular machines that interpret the sequences), and we can calculate an entropy bound as well.
Shannon did more than determine channel capacity. He gave a fundamental definition of what constitutes information. This applies to all information. That is why his work is considered groundbreaking.
In Eric’s fruit truck example there are more than one receivers and more than one set of information. To the woman who calculates truck gas consumption, the information is the weight of the load and the length of the route. To the person who loads the truck, the information is how much space is left. To the person who unloads the truck it is how much more is left to unload before I can take my coffee break. To the customer, it is what is wanted.
To the gas purchaser Mr. Shannon has maximized the truck’s capacity, but by only delivering a small subset of what the customer wanted, Mr. Shannon’s delivery is NOT what the truck is capable of, and he fails in his mission to maximize channel capacity for information as defined by the customer.
The truck is a special case and therefore kind of a poor example because it is capable of delivering, unchanged, a complete load of exactly what the customer wants without corruption. There is no ‘noise’ that corrupts the shipment(except maybe spoilage). We usually presume that the sender only sends the things the receiver (customer) wants and that something bad happens to the messages along the way. In biology, this is usually considered to be the DNA mutations that creep in and do nothing or cause problems.
Neil @2:
Well, which is it, the information or the channel capacity?
The only way you can say the metric is measuring “information” is to redefine information to mean the same thing as “channel capacity.” In which case we have then lost the value of the word “information” and we need to come up with a different word to describe what we originally used to think of as information.
The whole term “shannon information” causes more confusion than light. That is precisely why an analogy is helpful for people to start grasping what we are dealing with.
Again, as I said, I’m not going to change decades of people referring to the shannon metric as “shannon information.” That is too ingrained and there is too much inertia.
But we need to be very clear when talking about shannon so-called information that we are not talking about the key aspects that we typically think of when we use the word information. We are not talking about semantics, vocabulary, meaning, intent, purpose, informing. All those, as Shannon himself clearly stated (and as UB pointed out above) are irrelevant to the shannon metric.
Then you do not understand specification, as it is used in the intelligent design context. Specification involves meaning, purpose, function — precisely the kinds of things Shannon said were irrelevant to his channel capacity problem.
The argument from information sure has come a long way on UD. Years ago on UD I remember many neo-Darwinists denied information was even in the cell. In fact in the past, this following comment has been used far more than once on UD,,,
Then after a while, when some of the Darwinists’ realized that information actually was in the cell, and that it was not an ‘illusion’, their arguments ‘evolved’ to say that the information in the cell was merely Shannon information, and therefore, using Shannon’s broad metric for information, information, the Darwinists held, can therefore increase with Darwinian evolution,,,
But even using the ‘loose’ definition of Shannon information, instead of using the more precise definitions of functional information, (i.e. CSI, prescriptive, etc..), has insurmountable problems for Darwinists.
Moreover, using the Shannon metric for information (i.e. channel capacity), we find the first DNA code of life on earth had to be at least as complex as the current DNA code found in life is:
The reason why a code is a all or nothing deal that cannot be evolved gradually is best summarized by Dawkins:
And this problem of multiple codes, that can’t be derived by gradual processes, has recently become much more acute for Darwinists:
supplemental notes:
Responding to UB’s comment.
Sorry, but you are confused here.
The actual engineering that Shannon mentions, mostly consists of specifications and of hardware that implements those specification.
This is not the ID kind of specification, as in “Oh, it looks specified, so it just must be.” These are detailed engineering specification that you can find in libraries and probably on the internet.
A varying electrical signal can be converted to a sequence of bits in a gazillion different ways. But, somehow, the Internet works because we always manage to get the bits that were intended. It works because of all of those engineering specifications, and because the equipment all follows those specifications both in generating the signal and in interpreting it.
GBDixon @7:
Thanks for your thoughts.
Are you suggesting that if we know, for example, that we are dealing with, say, a 4-bit digital code, that it is impossible to calculate the shannon metric without first knowing all of the possible messages that can be sent using that 4-bit digital code?
I hope that is not what you meant by your first paragraph, because that is completely wrong.
No. He focused on one particular aspect of communication which is highly relevant to the transmission of information. That is why his work is considered groundbreaking.
Again, Shannon acknowledged that the meaning and the semantics — what we generally think of as information if you were to ask anyone on the street, if you were to look the word up in the dictionary, if you were to look at the etymology of the word — is irrelevant to his calculation.
Look, any analogy can be stretched too far. You are going off on tangents. The reason I worded the analogy the way I did is to focus on the carrying capacity, which is what the Shannon metric deals with. Don’t get off on irrelevant tangents about the driver, the coffee breaks for the dock workers, etc.
Yes, there is lots of “information” about a physical system that can be gleaned by someone observing the system. That is separate from the information contained in the transmission itself. (This question about information being contained in a physical system has been dealt with in detail previously on this site.)
Focus on (i) the size of the truck, (ii) the sizes of the different packets that can be carried on the truck, and (iii) the contents of the packets. Those are the key.
Yes, noise is an interesting issue, but it is easily included in our example if you want (spoilage in transit, box falling off the truck, someone stealing fruit from a box while the driver is parked at the coffee shop, etc.). Noise is an important point for transmission and one that Shannon spent a lot of time focusing on.
However, I would note that most of the calculations of the Shannon metric that are done in the context of the design debate assume a noiseless channel. Indeed, we need to assume a noiseless channel initially to come up with a base calculation (as Shannon also did). Noise makes the question of transmission more challenging, but it is not typically relevant for the encoding aspect we are thinking about.
Responding to EA’s comment (#8):
Strictly speaking, the metric is the amount of information. The channel capacity is a rate (max amount of information per unit time).
No, we have not lost the value of the word “information.”
As Shannon has said, information can carry semantics. But, as defined by Shannon, the term “information” does not refer to the semantics.
I understand that you want “information” to actually refer to the semantics. I used to think that way, and I probably posted comments to that effect on usenet, maybe 20 years ago. I have since come to recognize that I was mistaken. I now see that Shannon information, which does not refer to the semantics, is the most appropriate meaning for “information.” And, here, I am talking about what is appropriate for studying questions of human cognition. I’m not limiting myself to digital technology.
The problem with semantics (or meaning), is that it cannot be specified. If you want to insist that “information” is a reference to semantics, then you must give up on the “specified” part of CSI. It is the nature of ordinary language, that meanings are inherently subjective and unspecifiable.
A few related notes that may be of interest to the reader:
Measuring the functional sequence complexity of proteins – Kirk K Durston, David KY Chiu, David L Abel and Jack T Trevors – 2007
Excerpt: We have extended Shannon uncertainty by incorporating the data variable with a functionality variable. The resulting measured unit, which we call Functional bit (Fit), is calculated from the sequence data jointly with the defined functionality variable. To demonstrate the relevance to functional bioinformatics, a method to measure functional sequence complexity was developed and applied to 35 protein families.,,,
http://www.tbiomed.com/content/4/1/47
At the 17 minute mark of the following video, Winston Ewert speaks on how functional information is measured in proteins:
Proposed Information Metric: Conditional Kolmogorov Complexity (Ewert) – July 2012 – video
http://www.youtube.com/watch?v=fm3mm3ofAYU
=============
Conversations with William Dembski–The Thesis of Being as Communion (The Metaphysics of Information) – video
https://www.youtube.com/watch?v=cYAsaU9IvnI
Good grief Neil.
Let’s run this down:
Eric made a comment about the specification inherent in information – the “aboutness” of information.
You then turn around to object, saying “no” all Shannon information is specified, ‘noise isn’t considered Shannon information.’
I then point out to you that your comments are in direct opposition to what Shannon himself is saying regarding the “aboutness” of information (it’s specification) – i.e. its “correlation according to some system with certain physical or conceptual entities”.
So now you turn around and say the “specification” you’re talking about is about the system instead, not the information itself. Moreover, you want to pretend that when Shannon was discussing the selection of a message for its specificity (its “correlation according to some system with certain physical or conceptual entities”), he talking about the system specification as well. You are just deeply confused.
Finally, after having bastardized what Shannon has said, you want to use the opportuinity to take a juvenile slap at ID for having an appropriate interest in the aspects of information that are “correlated according to some system with certain physical or conceptual entities”. Obviously, you take a shot at ID on an issue you clearly do not understand.
Give it a rest.
Hi Eric,
In your first example of a four bit code:
The sender and receiver decide 1001 and 0110 are valid messages. This is the information. The sender only sends one of these two messages when she sends.
The channel possibly corrupts the messages into any of the other possible fourteen combinations of four bits, or leaves the messages the same. the fourteen combinations are not found in the message list and are NOT information.
These invalid messages plus the valid messages are the set of all possible four bit combinations that can be received and with these two parameters we can calculate the entropy of the information. Without an agreed-upon message list and the number of invalid messages that are possible, entropy cannot be calculated.
Note that we could have chosen any combination of four bits as a valid message. This is what Shannon means when he says the content of the messages do not matter. We may attach meaning to a message (“launch the water balloon”) but as fundamental information, only what is in the valid message list is information according to Shannon.
I attach semantics and go off on tangents to illustrate information depends on sender and receiver context. Here, the 1001 and 0110 messages are meaningless to us, but we can assert they are information because they are in the valid message list. They presumably have pithy meaning to both the sender and receiver but we as channel engineers are not privy to that meaning and don’t care what it is: only that 1001 and 0110 are valid messages and the rest are not (we discard invalid messages and ask for a resend or do something else).
Attaching semantics to valid messages does not alter any of the calculations or what constitutes a valid message but if we wish to know why 1001 is a valid message, we must consult the sender or receiver and find out what the message means. We have generalized the concept of information, as Shannon did (Shannon actually built on the work of Hartley).
Shannon did indeed generalize and formalize the concept of information. I think you may be stuck on Shannon’s channel capacity theorem, but he generated two fundamental theorems: the other is called the “source coding theorem” and deals with what information is.
Many very smart people had trouble with Shannon’s concepts at first. I recommend John R. Pierce “An Introduction to Information Theory” for a good explanation. It is cheap at Amazon.
Eric says:
Focus on (i) the size of the truck, (ii) the sizes of the different packets that can be carried on the truck, and (iii) the contents of the packets. Those are the key.
I agree with this as it relates to channel capacity, Eric, but aren’t we mainly talking about what constitutes information and not how to send it?
Where we appear to disagree is the idea I assert that the receiver (and usually the sender) determine what the information is and what it means. Regardless of what they choose, the information can be distilled into a set of messages that are generic in nature and are sent across a possibly noisy channel to be received, checked for validity, decoded and interpreted by the receiver (again, in cells the receivers are the molecular machines that interpret the DNA sequences). Information cannot be defined without a receiver to interpret it, and information changes with receiver context.
Neil- a definition of a word specifies that words meaning. And the entire world uses the word “information” in a way that information = meaning. Textbooks are full of meaningful information.
Shannon did not care about meaning because, guess what, the machines that transmit and receive the signal don’t care about it.
GBDixon:
Thank you for your clear summary of some basic concepts about Shannon’s theory.
I would like to mention here how the concepts of Shannon are applied in Durston’s model. There, the random state of a protein sequence of a certain length is considered as the highest uncertainly (highest H). The contraints given by the functional state (derived by the alignment of the known variants of that protein in the course of evolution) determine a reduction of uncertainty, which corresponds to the functional information in the protein sequence.
In general, for a protein sequence, the total number of sequences which exhibit the molecular function is the “list” of meaningful “messages”. Their meaning is their functional activity, and can be defined independently. In itself, the function has nothing to do with the calculation, except for its constraints on the sequence and the consequent reduction of uncertainty. So, an AA position which always remains the same will give a reduction of uncertainty of a little more than 4 bits, while an AA position which can assume randomly any value will give no reduction of uncertainty (0 functional information). All intermediate conditions can be found. The sum of the functional information at each position is the global functional information of that protein.
Replying to UB’ comment (#14):
Aboutness (intentionality) is not specification.
Again, aboutness is not specification.
I don’t recall Shannon mentioning “aboutness”, though he did mention meaning and semantics.
A specification ought to be specific. Meaning is never specific — it is always subjective and dependent on subjective interpretation.
Specification and correlation are not the same thing at all.
When a topic claims to be about Shannon information, then it ought to be about Shannon information. Meaning and semantics, in the ordinary language sense, are not any part of Shannon information. Typically, Shannon information is a sequence of symbols. Specification, in the context of Shannon information, can only mean specification of symbols.
If you wanted to discuss something other than Shannon information, this thread seems to be the wrong place for that discussion.
Responding to Joe (#16):
A definition of a word is an entry into a circular chain of dictionary lookups which never get you to a specification of meaning.
Neil Rickert:
Yes, Neil, we already understand that you choose obfuscation rather than education.
If we didn’t define words, ie if words were not specified, communication would be impossible. Definitions are word specifications.
And from data:
Successful communication was going on thousands of years before there were definitions or dictionaries or written language.
Bird manage to communicate with their bird songs. Where are the definitions that you claim would be needed?
Neil, you made a claim about Shannon information that is patently wrong. You were corrected. Shannon information does not consider any specification in the communication channel, consequently, unspecified noise in the communuication channel is included in Shannon information – just as Claude Shannon states in the second paragraph of his paper.
Neil:
Let’s not get all hung up on a definitional battle. I’ve already said that if someone wants to use the term “Shannon information,” fine. I’m not going to change that unfortunate term this late in the game.
But they need to understand that when they use the term “Shannon information” they are not talking about meaning, or content, or specification, or function. Shannon himself makes this quite clear.
We could call the shannon metric something like “statistical information” if people insist on calling it “information” at all. And we could call the other stuff “substantive information.” The particular definition doesn’t matter. As long as people are clear that there is a real, fundamental, substantive distinction between the two.
Gpuccio
The protein model seems very reasonable. Thank you.
Neil:
Take the following string:
ETTTBSNHOBOORTUEHTISEEATQTNOIO
What is the Shannon information?
This is not any kind of trick question. I’m sincerely trying to make sure we aren’t talking past each other and that I understand your point. Assume the foregoing is based on a 26-letter alphabet, all caps, no other symbols.
I think we’re on the same page that meaning and content are not part of Shannon information, so I want to see if we can get on the same page as to what is Shannon information.
Thanks,
Neil Rickert:
The context was communication using words, Neil. But thanks for proving you prefer obfuscation over education.
How do you know they are communicating?
Neil Rickart at #12 said:
In other words he agrees with the OP exactly as written. Thanks for the confirmation Neil!
This is plainly wrong.
Shannon was concerned about noise, and does not count noise as information.
Suppose I have a random generator, set to generate random strings. I take the output and put that up on a web page.
If you read that web page, you will that particular string, exactly as specified by my random generator. You won’t see a different string from what was specified. Maybe it looks like noise to you, but it is what the random generator specified. And the theory is concerned with transmitting that correctly (i.e. as specified).
There’s a lot of electrical noise around. But you still see the string that my random generator produced (and thereby specified). You do not see any sign of the noise.
People were succesfully communicationg with words thousands of years before there were written languages or definitions or dictionaries.
The string itself is the information.
If you intended to ask about the amount of information (the metric), I guess that’s about 141 bits (the number of letters, times the log of 26 to the base 2).
Neil Rickert:
So those words had no meaning then. What, exactly, were they communicating with those meaningless words? And if words don’t have any meaning how can you tell if they are words?
We are right back to:
If we didn’t define words, ie if words were not specified, communication would be impossible. Definitions are word specifications.
The definitions don’t have to be written down, Neil.
Not only do words have to be defined, but ball-park definitions, at the very least, have to be agreed.
As a matter of fact, agreement/usage is paramount, much to the chagrin of the Academie Francaise. So, you could scarcely be more comprehensively mistaken, Neil.
– facepalm –
Neil now wants to hook up a random generator to the input channel of the system in order to “specify” randomness to the output. By doing this, he has concocted a scenario where he can save his claim that “all Shannon information is specified”.
.
(I believe I’ll move along now)
I’m really curious at what level the OP understands the subtle implications of information theory, me thinking probably he doesn’t. Conversely, it seems the OP thinks that information theory is an attempt to conform to some function that he thinks it should but but somehow comes up short in its founder’s aspirations to achieve, maybe because of the misunderstanding of the subtleties.
For example, because of the OP’s little scenario it seems to me that he thinks there should be some type of quality factor attached to information, as if an artistic work could somehow be measured and shoveled in there. Or somehow the qualities of decision making by the young manager on the job. The original OP seems also to be confused by what Shannon was referring to as capacity. In a channel operating near BIT RATE capacity, the error-correction codes are embedded such that the message can be recovered WITHOUT ERROR due to the employment of the error-correction code (redundant information). It is the amazing result of the Shannon- Hartley theorem that it takes into account the success of any error-correction code and is one of those mysterious results from applied math that the theorem MUST take into account the action of error-correction whether or not the originators of the theorem intended this to be the case. And if you read the 1948 paper as I have, Shannon discusses this in entertaining terms. All communications will produce errors, but the amazing thing is that error correction can reduce the errors to zero as long as the recovered information bit rate does not exceed that specified by the Shannon-Hartley theorem, irregardless of the efficiency of the error-correction code employed. (efficiency measured as the numbers of errors corrected per second per additional bandwidth required for the redundant bits for the error correction code)
1.) Shannon information is information.
2.) Shannon was completely correct to not refer to it as a measurement or a metric, because what he explicitly denies is that it tells you whether or not you are measuring information!
3.) People who deny #1 are mistaken. They fail to understand what Shannon information is information about. It is explicitly not about measuring information!
4.) People who do not understand #2 and #3 often mistakenly claim that Shannon information demonstrates that information can be meaningless.
We get our fair share of both here at UD. Allen MacNeill and Elizabeth Liddle spring to mind.
But meaningless information? Really? How does anyone get that from Shannon’s theory? Please. Speak up.
So, Eric, thanks again for treading where angels fear 🙂
Questiosn for all to ponder:
Q1: What is Shannon Information?
Q2: What is Shannon Information about?
Q3: Is Shannon Information independent of the information content (or lack thereof) of the message?
Q4: Does Shannon Information tell us whether the message is meaningful or not?
Q5: If Shannon Information cannot tell us whether the message is meaningful (or not) can it therefore be deduced from Shannon’s theory that information can be meaningless?
Q5: What would “meaningless Shannon Information” look like?
Mung:
Warren Weaver, one of Shannon’s collaborators:
What is information without meaning if not meaningless information?
Don’t mean to pick on you Neil, because I haven’t yet read all your comments in the current thread, but you make some fundamental errors which will hopefully be instructive.
No, it isn’t. Shannon’s theory does not tell you whether this message constitutes information or not.
Is that assuming an alphabet of 26 symbols with each having an equal probability? IOW, not English?
I say assuming, because how did you know? Did Shannon’s theory tell you?
Eric Anderson:
That may not be a “trick” question, but it is a meaningless question. In the English language there are a specific number of letters and certain letters appear more frequently than others. Is that a message in the English language? I think not.
So how then are we able to reduce the uncertainty?
Neil @31:
Thanks. Let me make sure I’m understanding you.
With a string, say, ETTTBSNHOBOORTUEHTISEEATQTNOIO, you are saying that the string itself is “Shannon information.”
Presumably, then, the string HTISEEATQTNOIOETTTBSNHOBOORTUE would also be “Shannon information.”
And the string HTISEEATQSNHOBOORTUETNOIOETTTB would also constitute “Shannon information,” and so on.
In other words, under your definition any string of characters constitutes “Shannon information.” And a single letter and a whole page of characters or a whole book also each constitute “Shannon information.”
That being the case, perhaps you can clarify for me what the difference is between Shannon information and strings of characters? Under your definition they seem to be equivalent. If you are right, then we can say:
Shannon information = string of characters
Which is to say, if we have a string of characters, then the Shannon information concept tells us that we have . . . a string of characters.
What does the term “Shannon information” bring to the table in your definition, if it simply is another way of saying that it is “the string itself”?
Mung:
Thanks for your comments. I definitely want your input, as you have been very vocal about this for a long time. I do trust that after we’ve had a chance to mull on your questions for a day that you’ll treat us to an actual explanation and not just let us flounder at sea while you ask provoking questions. 🙂
It is not meaningless at all. I’m trying to step back to square one to understand what Neil is referring to when he talks about “Shannon information.” The example is very much on point. Yes, I could have said the letters had equal probability if we wanted to make the calculation simpler. That impact on the calculation is not directly important, however, to the question I posed. Hopefully, my follow up @39 will help flesh out precisely what Neil views as “Shannon information” so that I can understand it and be on the same page.
Joe:
Give me an example of meaningless information.
So when he says information in the second sentence you don’t think he means information [as used] in this theory … in a special mathematical sense?
So why should information have a special sense in this theory but be nonsense elsewhere?
If information can be meaningless, why is Shannon Information exempt? Is Shannon Information not information? Do you think he means Shannon Information can be meaningless?
All Weaver is saying is what I said previously. Shannon Information is mathematical, it doesn’t tell you whether whether or not the message is meaningful. It does not logically follow that information can be meaningless.
In fact, it is not logically possible for information to be meaningless.
But to the specific challenge I raised, which you did not address. How does one get “meaningless information?” from Shannon’s theory. Given that it’s a mathematical theory of communication, I expect to see some maths.
hi Eric,
When I said the question was meaningless, I meant that the question did not have an answer according to Shannon’s theory.
It’s like asking what is the Shannon information of the sky. Like asking whether a red sky has more or less Shannon information than a blue sky.
Yes, Shannon Information is a measure, but it’s a measure of probabilities, not a measure of information in the classical sense.
Let me try to give an example:
Say we could create a symbol generator that could generate an infinite number of different symbols each with the same probability. Could Shannon theory be used to measure the information in bits?
Say we could create a symbol generator that could generate a finite number of different symbols but with no predictable probability for any given symbol. Could Shannon theory be used to measure the information in bits?
If you have a “coin tossing” machine and the two symbols it generates are H and T respectively and they are equiprobable, then when you see an H you can say your uncertainty has been reduced by a certain amount, and we can call this “information.”
But if we randomly change the probability of the T or the H, or if we randomly insert other symbols, then how do we measure the probability, or the reduction in uncertainty, or the “Shannon Information”?
Cheers
The Shannon Information has degraded accordingly!
Neil Rickert:
The measure is not that which is being measured?
I could not agree more. Confusing the two is a source of much confusion.
Is it not also the case that if the measure is “an amount of information” it does not logically follow that it is information that is being measured?
Neil Rickert
It does not follow that because a measurement can be taken in “amounts of information” that what is being measured is information.
It also does not follow that because a measurement can be taken in “amounts of information” that what is being measured is meaningless information.
Neil Rickert
The amount of information of what?
All that’s being “measured” is the capacity to process symbols. The symbols may or may not convey meaning. Whether or not they convey meaning is irrelevant to the mathematical problem. From this, it does not logically follow that there is or can be such a thing as “meaningless information.”
GBDixon:
Neil Rickert:
Right. These are real specifications
But they have nothing to do with “the ID kind of specification.” Right.
Eric:
This is what Elizabeth Liddle thought. Toss a coin x number of times you get y bits of “Shannon information.”
The coin tosses were meaningless. When asked what they were about she had no answer. But because y bits of Shannon Information could be calculated it somehow meant that information could be meaningless.
Unfortunately for ID, that is also the view presented by Stephen Meyer in Signature in the Cell.
Eric:
Are we then in agreement that Shannon theory cannot tell us what is information and what is not information? That Shannon theory cannot tell us which messages are meaningful and which are not meaningful?
Because Shannon theory lacks this capability, does it in any way logically follow from Shannon theory that information can be meaningless?
Say the fruit truck confused Avocado with Almond. IF the symbol “A” could mean either one what are the implications?
Mung @41-47:
Whoa, slow down there cowboy! Let Neil get a word in edge-wise! 🙂 I know you’re already fired up about this!
@47:
Do you have a cite for this? I could maybe look it up, but I’m too lazy and I might not find the actual passage that you are referring to, so if you have a quote handy, that would be helpful.
If what Meyer is saying is that a string of characters can be meaningless, then I’d have to agree.
If what you’re saying is that a string of characters (given relevant parameters, of course) can contain a certain number of bits of Shannon information and that, therefore, the Shannon information is “meaningful” because it is “about” something, then I’d say that is trivially true . . . and also wholly uninteresting for purposes of ID, which is no doubt what Meyer was focusing on.
Anyway, if you have the quote handy, that would be great.
Strictly speaking, it is Shannon information if it is part of a communication system or an information processing system.
Marks on paper that just happen to look like letters would not count. This is why it is reasonable to say that Shannon information is always specified (by being entered into a communication system).
Hmmm . . . Slipped in a #48 while I was typing, eh?
Well, that depends on what we mean by information. I’m inclined to think that information has to be meaningful, but other folks (as we have seen), argue that any old string of letters can be information. That is part of the definitional confusion that results from the term “Shannon information.”
Quite true. Shannon himself said as much. Indeed, he used the word “irrelevant” when talking about the meaning of a message in the context of his theory.
Again, this is a definitional issue. Shannon theory simply cannot say whether we are dealing with meaning or not.
So if information necessarily has meaning, as you appear to be arguing (and with which I would provisionally be inclined to agree), then — by definition — Shannon theory cannot tell us whether we are dealing with information or not. At least not in the underlying substance.
Thus the so-called “Shannon information” must be about something entirely different than the substance of the message, the substance of the string, the substance of the communication. The metric is measuring something separate from and apart from the underlying information.
That is part of my whole point.
Mung (#38):
Strictly speaking, that is correct. It is a mathematical theory. One chooses whether to use the theory to model an actual communication system. However, people find it more convenient to say “x is Shannon information” than “x can be modeled as Shannon information.”
Neil @50:
Thanks.
Just to be clear, are you saying that something entered into a communication system has to be a valid communication (i.e., meaningful, understandable, purposeful, or whatever similar term we want to use here)?
Such underlying meaning would indeed be a specification.
Or are you arguing that the simple act of entering a string into the system (whether or not that string has any underlying meaning or substance) makes that string a specification?
Eric (#51):
This may be the most important point of the whole discussion. The word “information” is used in multiple conflicting ways. Unfortunately, people talk is if information were an objective or metaphysical entity. We would do better to understand it as an attribution appropriate for a particular use.
No. It is best to leave those ideas out of the issue.
A video camera transmitting data could still count as a communication system, even if it is entirely mechanical and no part of the system has a capacity for meaning or understanding.
You can look at it as in my reply to Mung. What makes something information, is our choice to use Shannon’s theory to mathematically model it. Your string of letters is Shannon information, in the sense that I took you as communicating that string in a way that can be modeled as Shannon information. However, in completely objective metaphysical terms, your string of letters is ink marks on paper or illuminated dots on a screen.
The reason I like Shannon information, is that it gives us that stark picture that nothing actually is information. Rather, we count something as information based on how we use it.
And sure, we have people claiming that the Universe itself is just information. Whatever it is that they mean by “information”, it isn’t anything that I can make sense of.
It’s best to leave whatever Neil sez out of the discussion as it has become very obvious that Neil doesn’t know jack about information.
BTW marks on a paper that look like letters could very well be information. As I said, Neil is the wrong person to discuss information with as he thinks communication can take -place with meaningless words- whatever those are.
Hi all,
This will be my last comment. I have irritated our host and come off as a know-it-all. This is important to me, I guess. I think that a lot of confusion, argument and new vocabulary could be avoided if ID advocates knew information theory a little better. Perhaps this summary may help a little:
Shannon divided the information transmission problem into two pieces: the encode/decode problem and the transmit/receive problem. His famous channel capacity theorem is the solution to the second problem and is the domain our host and most others have been working in.
But the nature of information, what it is, and how to best represent it is the subject of the encode/decode problem. Here, information is defined, mainly by the receiver, and meaning is the most important aspect of the problem.
In his encode/decode theorem, Shannon showed how a message with redundancy (an example of redundancy is how u nearly always follows q in English) could be encoded into the minimum-sized message possible (“pure” information if you will). We are all familiar with the technique: zip and tar files reduce size by removing redundancy. In this first problem domain information is simply what informs the receiver. Nothing more or less.
Once the information is reduced to its smallest generic form, it is prepared to be sent through the channel and we enter the second problem domain. Here, a special form of redundancy (error correction codes) are added to the generic messages and they are transmitted. In this domain the meaning of the messages is irrelevant, thus the comments to this effect.
But once the messages are received they are decoded and meaning is once again assigned to each message. We are back into the domain where meaning matters.
It is as simple as that: information is the stuff the receiver sees as information. Complex patterns in mud do not inform us much…they don’t contain much information. But we have learned a great deal from DNA. It is tremendously information-rich to both us and the molecular machines that process it.
We are privileged to live in a time when the Shannon channel capacity has been practically reached. We have technology that is fast and cheap enough to transmit information error-free at essentially the maximum theoretical rate.
I believe the entire universe was set up so we are able to figure out, in every detail, exactly how it was all done. A magnificent classroom or lab, if you will. What marvelous discoveries await us!
information [in-fer-mey-shun] noun
1. knowledge acquired through experience or study
2. knowledge of specific and timely events; news
3. the act of informing or the condition of being informed
4. in computing: (a) the meaning given to data by the way in which it is interpreted; (b) another word for data
Synonyms: data, facts, intelligence, advice.
—–
Neil suggests @55 that it is best to leave things like understanding and purposeful informing and meaning “out of the issue.”
In other words, if we strip the word “information” of everything that makes it information, if we strip it of all ordinary meaning (no pun intended), then we can use the word to mean something else than what is normally meant. Shoot, if we’re going that far, why not just use the word “dogs” instead? We could strip it of all its normal meaning and say that we are now dealing with “Shannon dogs.” Just as rational.
Sorry. Shannon was a great guy, but he doesn’t get to completely strip the word “information” of everything that the word has meant for hundreds of years and say, in effect, “anything that can be modeled with my theory is information, everything else isn’t.” Furthermore, Shannon never made any such claim.
Again, I don’t begrudge anyone using the term “Shannon information” as long as they are clear what they are talking about. So far, it is wholly unclear what you mean.
Wait a minute. Any string, every string, can be modeled with the Shannon metric if we know the relevant parameters. Previously you said it was all information. Now you’re saying nothing is?
Oh, I see. We count something as information if it is usable to convey something of substance, if it has meaning, if it informs. I certainly agree with that. And it takes us right back to the old dictionary definition.
—–
I think we probably have the same uneasy skepticism here. Not real clear to me either, though I do understand the general thrust of information being at the heart of something or being the source of something.
GBDixon- Thank you for your posts- You haven’t irritated me and that says quite a bit because when it comes to information this information technologist is a know-it-all who would have been irritated had you posted nonsense. 😉
BTW, Neil, getting back to the question @53:
Since you answered ‘no’ to the meaning option, I take it you are saying that the simple act of something being entered in a communication system makes it a “specification.”
We could, I suppose, refer to the mere sequence of any string as a “specification”, which is really no more substantive than saying that the sequence is what it is. In other words, we would be saying that the sequence equals the specification and we could just use the word “sequence” instead of “specification”.
Yes, we can refer to the sequence of a string as its “specification,” and that might make sense as a shorthand in some circumstances. That is a purely descriptive sense, and (as Shannon mentioned) is devoid of any concept of substance or meaning.
But we need to keep in mind that this is different from the “specification” concept that design advocates talk about. In this latter case we are very much concerned with substance and meaning and purpose and function.
—–
This all really underscores the broader point of my post. Namely, the concept of Shannon “information” is not really helpful or relevant to the key issues of specified information and design. Shannon information is helpful and relevant only for the base identification of “complexity” in certain circumstances. So it can help with the “C” of “CSI”, but is not really useful for thinking about the “S” or even the “I”.
You know really, you guys, at least some of you, I think you miss something really fundamental. The usefulness of information can only be surmised by a mind. Just because one of you throws out a string of characters and denies that it is information, that denial itself can be determined to be information, or denied to be information.
For example the length of that string might be an indicator in how much of a hurry was the contributor. So when Eric A typed out his two strings above, he thinks they mean nothing. But I have a mind, strangely enough, and I can tell that there was a high probability his caps lock key was on during both of his strings. This is information. Also, since I can see that there were no spaces and no significant wording to his strings, I can discern something of his intent, and his intent was to post gibberish for part of the post and to use it for didactic purposes. Especially since he did not reveal in the post any real world process by which the strings were created, even when ignoring all of his philosophical points. This is information. He did not type any punctuation characters, and so I can tell that he was biased against a significant subset of the ASCII code, admittedly I don’t know why or if it was just a temporary bias, but that is more information.
And actually Eric has never even said anything about why, when he types a meaningless string, why we are all confident that what he keyed is what we all see on our screens, because none of us can check for spelling on those strings. And so that tells me that Eric is confident that all of his keyed input appears with no errors on all of our screens, without speculating on why that is, after reaching each of us via a unique set of channels. And that his “meaningless” strings did transfer error-free and did not appear with a couple of non-trivial words to one or two of us because of errors. Eric’s confidence in that error-free transmission is more information about him, but I believe there is little probability he understands why it is error-free and that is information.
GBDixon @57:
I hope you’re not referring to me, as I certainly haven’t been irritated by your comments. I appreciate your thoughts.
Meaning definitely matters to the normal concept of information, and I think your breaking down the communication process is helpful.
A separate question is whether there is information independent of the recipient. We have discussed this in some detail previously and, I think, concluded that information arises at the recipient side — by definition prior to the time it is encoded.
Later, if there is a recipient who can receive and understand the information that has been encoded, great. If not, it doesn’t necessarily mean that the information doesn’t exist. It is still there and could be discovered later.
For example, the information encoded in DNA was there — objectively functioning and doing what it was supposed to do — for millennia before the genetic code was discovered.
GBDixon,
I hope you come back. Your explanation at 57 was one of the best I have seen on this site. I usually tune out when someone discusses Shannon information because it is usually obtuse and then there is the contentious bunch who always want to create noise and prevent communication.
So reconsider. Your posts have been a breath of fresh air.
groovamos @61:
With respect, you are getting caught up in irrelevancies.
As an intelligent being, are there things you can ascertain or discover or infer about some object in the real world (whether a rock at the seashore or a string of characters on your screen)? Sure. Lots of things. You can infer that I wrote my string with Caps Lock on, on a computer, that it was transmitted without error, etc. Those are things you can ascertain — pieces of information you can discover — about the string.
All of that is 100% irrelevant to the question that was posed: Does the sequence of characters itself contain information?
We have addressed in detail previously on this forum the difference between (i) an observer being able to discover information about an object (what you are focusing on), and (ii) representative information encoded in or contained in the object.
It is critical to distinguish between these two. Your comments are exactly on point for (i). But they are completely irrelevant to what we were discussing, which is (ii).
jerry @63:
LOL! Great use of Shannon terminology! 🙂
Eric: I think it not irrelevant at all to point out that it requires a mind to determine relevance, but that is my opinion, maybe not yours. You intended for your string to be irrelevant. It is obvious that it was your intent, and whether or not it is relevant what your intent was to the study of statistical communications, is really a matter of personal opinion, which requires a mind.
An anemometer connected to a telemetry system sends back what to you would likely look like “random” bit sequences. With correct knowledge of the formatting and coding ( a big chunk of information in itself) the bit stream can be converted in to a sequence of numbers in “real” time, possibly very useful information. A researcher tries to fit the numbers to a statistical curve and concludes that they indicate a Rayleigh probability density for the velocity. This might invoke from you a “so what” response. If you had another piece (a well-known one) of information in your mind, you might instead say “Oh the wind speed with direction using any orthogonal coordinate set is a joint Gaussian probability with equal variances”.
So at different point along the communication/information chain your ability to make sense of an otherwise seemingly meaningless bit stream requires more information. How important the information is at any point of the analysis is obviously a matter of opinion or mind.
Eric@62,
Thank you, you are very gracious.
We have many situations regarding information, don’t we? Where there is a sender and receiver, they together establish what is information by a contract usually called a protocol. Our Ethernet, cell phones, etc. all operate under these various protocols.
You touch on what might be called latent information: we learn nothing from it at present, but we know it is there and if we only knew a little more it would become true information to us. The Rosetta stone is an example, I think. Here we had all these ancient texts…we knew they had valuable information but couldn’t decipher it. After the Rosetta stone was discovered that information became available to us. Here was a situation where there were senders and receivers but no contract. We had to figure out the senders’ protocols on our own.
The cell is similar. With each new discovery more information is unlocked. We know there is much, much more we do not yet understand. Because the sender is unknown to us, we are left to our own devices to figure out the protocols. I too believe latent information exists, that it is not true information to us simply because we do not know what it means, but that it will someday reveal many things. The ability of the cell to preserve seemingly worthless DNA sequences over eons hints that there are buried treasures we have yet to discover.
Is anyone else smiling. No? Okay…
🙂
Fantastic discussion.
I’ve hammered this home on numerous occasions…. the encoder/decoder problem….
And this alone makes Darwinian evolution the poppycock it is…..
Eric (#60):
I’m not quite sure of the point that you are raising there.
When you earlier asked about a string of letters, you called them letters. You did not call them pencil marks or illuminated dots on the screen. By calling them letters, you identified them as symbols rather than marks. And symbols, at least as I look at them, are abstract intentional objects rather than physical objects.
I’m inclined to say that there is already some specification going on when we take something to be a symbol.
Given that I have never been able to make sense of the “specification” that ID proponents talk of, I can readily agree that they different.
groovamos @66:
I don’t disagree with the idea that a mind must be involved with information. I have stated as much elsewhere and didn’t say anything to the contrary here.
I was simply responding (a bit harshly, I realize, and I apologize) to your suggestion that I didn’t know what I was talking about because there were all kinds of things that could be ascertained by an observer about the string, and how it came about, and how it was transmitted and so on.
It is not the case that I didn’t realize any of those things. I certainly realize them, have dealt with them in a prior post, and have moved on to a separate, more narrow issue in my exchange with Neil.
Specifically, the whole point of my string example was to try and ascertain from Neil what he thinks the term “Shannon information” means, since he had made the (rather strange to me) statement that the string itself is the Shannon information.
Please don’t read anything into my question to Neil beyond that. I generally agree with you that mind is quite relevant to the exercise of (i) producing information, and (ii) recognizing information. (Though this is not to ignore the very interesting question of whether information can exist objectively independent of an observer, which has been discussed on a prior thread).
At any rate, I apologize if my response was too harsh. I’ve addressed the basic aspects of information in other threads and in this one I am trying to home in on a couple of rather technical points due to the confusion some have (as evidenced by some of the comments on this thread) about what the Shannon metric can and can’t tell us.
GBDixon @67:
Thanks for the additional comments. Your example of the Rosetta Stone is precisely on point, and is an excellent example of information being there, available for the discovery. As is the information in the cell.
I think I’m largely in agreement with your thoughts on that point. The only nuance, perhaps, is that I would suggest that we need to acknowledge that information can exist independently of the observer/recipient. Indeed, if we take the view that information only comes into existence at the point of recognition, then we have inadvertently collapsed two separate concepts into one: the existence of information and the recognition of that information.
Anyway, I don’t mean to get into that here and go OT. We discussed it in some detail in a prior thread. I don’t think you had a chance to comment on the prior thread, but it might be interesting to read if you haven’t already, as it is very much on point with the issues you mentioned.
http://www.uncommondescent.com.....ion-arise/
Thanks again for the good thoughts.
Neil @70:
Thanks. Let me see if I can phrase the point about specification another way.
Take the string:
TOBEORNOTTOBETHATISTHEQUESTION
Now take the string:
ETTTBSNHOBOORTUEHTISEEATQTNOIO
Both of those strings have a “specification” as you are using the term, meaning that the string consists of a series of letters in a particular sequence. And if we were to write out that specification, we would simply be writing out the sequence. In other words, the way you are using the term “specification,” it simply means whatever sequence we happen to be looking at. And, yes, we can “specify” what sequence we are looking at by reciting the sequence. But that doesn’t add anything or tell us anything beyond the sequence itself.
However, it is also obvious, I trust, that there is something substantively different between the two strings. And that substance doesn’t depend on the number of letters, or the number of particular kinds of letters, or even the number that will get spit out of a simple Shannon calculator.
Rather, the difference lies in the fact that one string has underlying meaning, it has content, it specifies something (beyond the mere repetition of itself), it symbolizes something (again, beyond the mere repetition of itself). That is what we are talking about in the ID context by specification.
It is rather basic, but let me see if I can take a stab at it. A specification in the context of a string as in our example, is the function, or the meaning, or the symbolic representation, as I’ve mentioned above.
Again, take the two strings above. The second string can be described, it can be sequenced, it can be analyzed and calculated and poked and prodded and we can, through our faculties and tools, make lots of observations about that string.*
We can also do the exact same thing with the first string and make every single similar kind of observation or analysis about it.
And yet, when we are done with all that analysis, there is still something additional with the first string. Something symbolic, something meaningful, something representative — something that specifies something beyond itself. That is a specification for ID purposes.
It is not mysterious or unusual or tricky. Indeed, in most cases it is so obvious that we tend to gloss over it or don’t give it a second thought. As we go about our lives we are literally swimming in specifications, so we take it for granted. So sometimes I find it is helpful to step back to a simple example just to home in on what we are talking about.
Thanks,
—–
* Some people get confused and think that this means the string “contains” information. It doesn’t, as discussed in detail on the prior thread I referenced.
Neil Rickert:
Well I, for one, disagree that they are different. For example, Stonehenge- we can assume there was a design/ engineering specification, ie something that said what to build, what type of stones and where to place those stones, even though no one has ever found such a thing. The pyramids, same thing. The Antikythera mechanism must have had a design specification but no one has found one.
That said, IDists say we can identify a specification by the functionality and what it takes to get that functionality. If just about any configuration can produce the function then it isn’t that specified. If very few or only one configuration can produce it then we say it is specified.
This all goes back to Crick’s notion of biological information and the functionally specific sequences it takes for living organisms to produce the proteins they need to survive.
Eric: “I was simply responding (a bit harshly, I realize, and I apologize) to your suggestion that I didn’t know what I was talking about because there were all kinds of things that could be ascertained by an observer about the string, and how it came about, and how it was transmitted and so on.”
I did not suggest such. Your example of the quality of the cargo placed on the truck suggested that information theory maybe has a shortcoming in some philosophical sense. And maybe the philosophical sense is the only way you can go at it because neither you or anyone else brings error-correction to the table, I did, and then GBD also correctly ran with that aspect. So I take your example and where the wrong items have been shipped I suggest what theory can only contribute in the scenario and that is provide a foundation to correct the errors, and then when someone doesn’t like the taste of what was shipped, their displeasure in text form can be transmitted error-free, or over a voice channel with lossy compression at minimum. I was hoping you appreciated attention to your blog by a couple of people with expertise in information theory. My suggestion that you maybe don’t know the mechanisms of error-free transmissions was based on the fact that the topic was not broached until my post, when errors were at the heart of your proposed scenario. And that you obviously were counting on error-free appearance on our screens of your text. I might put up another post about the topic because I have a thesis project going with information theory at the core of it.
Joe @74:
“Well I, for one, disagree that they are different.”
Joe, just to clarify, what Neil was referring to is the fact that in order to run a string (it could be any old random string) through a Shannon calculation, you have to “specify” what the string is. That is what he was calling a specification.
As I mentioned to Neil, yes, it is true that you can “specify” a string by listing or reciting or entering into the system its sequence. It is true, but trivially true, and is not interesting for ID purposes. For ID, as you mentioned, we are interested in an underlying specification, a meaning, or function, or purpose, or representation.
That is why his use of the word “specification” and what ID is talking about are not the same.
I think that’s right.
BTW, it is also part of the confusion Dembski ran into in his paper, Irreducible Complexity Revisited, which I addressed in my lengthy and now-largely-forgotten response paper.
Eric- Got it. However Neil originally said that specification, in the way it is used in the design industry, is a set of technical writings and/ or drawings/schematics pertaining to the object/ structure/ event. See Neil’s comment in #10.
And that is why he doesn’t understand ID’s use. I was trying to explain that it is the same and we infer that specification from the calculated (biological) information/functional sequence specificity of whatever structure we are examining.
That also means we are removed from the world of mere intelligent agency involvement to actual intentional design, which is what ID is really after, IMO.
Joe:
Good catch on #10. I see what you are talking about.
1. The “specification” Neil refers to in #10 is indeed a design-based specification, and is an example of what we are talking about with ID.
2. However, the “specification” Neil refers to in #2 and #18 seems to be referring to the mere sequence of symbols that get plugged into a Shannon calculator or communication system.
#1 is substantive, and is what we are interested in for ID purposes (as you note).
#2 is merely descriptive, and is generally not what we are interested in for ID purposes.
groovamos @75:
Thanks for the further comments. I’m not sure we have any real substantive disagreement, but perhaps I can clarify the purpose of my post, in case it wasn’t clear.
I am aware of the issues regarding transmission and error correction. Had I wished to do a post on those topics or to give an example in that area, I would have. While those are important aspects of Shannon’s theory, to be sure, they are separate from the issue I am focusing on (and the one that so many anti-ID folks get hung up on), namely, the fact that the Shannon metric is not concerned with underlying meaning.
No. Errors in transmission were not at the heart of my analogy; they were irrelevant to it. My example assumes an error-free transmission.
Again, the question of error-free transmission, correction, receipt, and even decoding, are not what I am focusing on. I am homing in on a very specific debating point that is all too often brought up by anti-ID proponents: namely, the claim that the mere generation of complexity (which, given relevant parameters, can be measured and assigned some numerical value using the Shannon metric) demonstrates that genetic algorithms, natural processes and the like can generate “information.”
That is false. At least it is false in any meaningful sense and is only nominally true if we twist the definition of “information” to mean something without substance. Thus my post.
As to the other issues that don’t directly relate to the debating point I am focusing on (but that I heartily agree are interesting in their own right — things like error-free transmission), I would certainly love to see your post if you get a chance, and I’m sure I and many others can learn much from your experience and expertise.
Well, Eric, in a sense all of this is about error correction!
🙂
Eric:
LOL! ok, I just worked two days and two nights on minimal sleep. So Neil’s had his chance 🙂
Neil:
And this is the case because communications systems are intended to be used for communication and information processing systems are intended to process information.
I sure hope someone is understanding the fundamental error of (some of) the critics. What does it mean to say that you have communicated a nonsensical string of characters?
What happens to a communication overwhelmed by noise?
What happens to an information processing system fed a sequence of nonsensical bits?
Say you have a communication system and it is used to transmit a meaningless string of characters. Does it logically follow that communication can be meaningless?
communication
Neil:
Not really. *sigh*
Neil:
What does this even mean? You mean meaningless marks?
Wouldn’t the same hold true then for a sequence of meaningless symbols?
Neil:
Not really.
Nothing *specifies* Shannon information. It can either be calculated or it cannot be calculated.
What are the conditions that permit the calculation of Shannon information?
You’ve certainly hinted at them in your post. Can you transition into “teaching mode” for a bit?
Are there prerequisites for the calculation of the amount of Shannon information? What are they?
Eric:
Close. So close. 🙂
I agree.
Assuming agreement that Shannon theory simply cannot say whether we are dealing with meaning or not, then how can Shannon theory say whether we are dealing with information or not?
And if Shannon theory cannot tell us whether we are dealing with information or not, I say I have made my point. It is simply illegitimate to conclude from Shannon theory that information can be meaningless.
Eric:
I agree, though perhaps not for the same reason.
Even if it were in fact the case that information can be meaningless, how does the conclusion that information can be meaningless in any way follow from Shannon theory?
How does Shannon theory establish that we are in fact dealing with information?
Even if information does not necessarily have meaning, Shannon theory simply cannot tell us whether or not information must have or does have meaning. That is simply not in the remit of the theory.
It simply cannot tell us what information is.
Eric:
I agree! Whee!
It is “measuring” (calculating would be better) probabilities of symbols/messages given certain already understood information.
Shannon Information is not itself meaningless. Why should it lead people to conclude that non-Shannon Information can be meaningless? Where’s the logic?
I’ve argued all along that information is about something and that this is true as well of Shannon information.
It’s commonly accepted that Shannon Information is information, but less well understood (by far) what it [Shannon Information] is information about.
Questions?
Neil Rickert:
That would also include “Shannon Information.”
GBDixonJuly 1, 2014 at 7:51 am
A shame. Even a know-it-all generally knows something!
For what it is worth I had hoped to hear more from you.
GBDixonJuly 1, 2014 at 7:51 am
You see, I agree completely. But why would you say this then disappear? Don’t pontificate, educate!
Many of us here at UD are utterly receptive to learning new things. This is especially true of information theory.
Please stay.
Responding to Eric (#73):
Right. Shannon’s theory is a theory of communication, so specification ought to mean the specification of what is to be communicated.
No, not at all. There’s a difference in how we react to them. But that’s a difference in our reaction, not a substantive difference in the strings.
No, that’s nonsense. Neither string has any meaning. We contribute the meaning. Meaning is not a property of a string. Rather, meaning comes from us. Meaning is inherently subjective. But none of that subjective quality that we call “meaning” is ever present in the communication channel.
If you want an objective science of communication or of information, then you must leave meaning out of that science.
Neil Rickert:
Bullshit!. If Shannon Information isn’t meaningful, then of what use is it?
Mung @83:
There is no question that the Shannon measurement/calculation is about something.
The big problem comes because people think they are calculating the amount of “information” in a string when they run the Shannon calculation. They aren’t. Indeed, the only thing they are doing is plugging some parameters into a formula and producing a measurement or a calculation result. It is very unfortunate that this measurement has come to be called (by some people) “Shannon information” as many people then trick themselves into thinking that the measurement has somehow quantified the amount of “information” in the string.
We might as well take the temperature of an object in degrees Kelvin and call the result “Kelvin information.” Or we could plug some numbers into Einstein’s most famous equation and pronounce the result “Einstein information.”
Of course we will have produced information when take a measurement and assign it a value according to an agreed-upon system, or when we plug parameters into an equation and produce a result. Yes, that result is “information.” Yes, as a result of our mental faculties, in concert with accepted linguistic and mathematical conventions, we produce new information when we express the results of a measurement or when we set forth the results of a calculation. Those resulting numbers are information in their own right. But it doesn’t mean those numbers represent a measurement of information.
The resulting numbers themselves could be plugged into a Shannon calculator to yield a number. And that number is of course itself another piece of information. Which can then be fed back into the Shannon calculator itself — just like any other piece of information — to yield another number. And the cycle can continue . . .
The problem — what is really a very simple conceptual problem, but which has caused nearly endless difficulties for some people — is that (unlike in nearly every other calculation that we run in any other field), some people insist on calling the result of the Shannon calculation “Shannon information.” Couple that with the fact that information is generally plugged into the Shannon calculation on the front end, and many people get quite confused into thinking that the “Shannon information” coming out the back end somehow represents a measurement of the amount of “information” going in the front end.
It is easier to keep things separate in a case like Einstein’s equation, for example. We can see that if we take mass and multiply it times the speed of light squared then we will get a number that represents energy.
Unfortunately, in the Shannon case, people see “information” plugged into one end and “Shannon information” coming out the other end and think that this little number must somehow represent the amount of “information” that was contained in the input string.
At any rate, the upshot of all this is that the term “Shannon information” is a tremendous stumbling block, particularly for individuals who are approaching the topic (hoping to discredit the design inference) by claiming variously that (i) everything contains information, or (ii) information can be meaningless, or (iii) the Shannon number measures the actual information content, or (iv) some combination of the foregoing absurdities.
Yes, those of us who are a bit more careful can keep in mind what “Shannon information” really means and what it is really about and can avoid getting confused. But it is a very tough row to hoe to try to explain the issues to someone who is already entrenched in misunderstandings and who, unfortunately in so many cases, has a mental or philosophical predisposition to not understand the issues.
Neil @86:
Again, all your “specification” means is the sequence of the string that is plugged into the Shannon calculation. Yes, we can call that a “specification,” but it doesn’t have anything to do with a substantive concept of specification, which is what ID is about.
Of course an intelligent being was involved in assigning meaning to a string. And because there is a symbolic connection between the real-world meaning and the string, we recognize that the string contains representation, meaning, substance. And we also know, just speaking of ID for a moment, that such a situation is only known to arise by the activity of intelligent beings.
If you’re arguing that meaningful information/communication doesn’t arise without the activity of an intelligent being to provide that meaning, then I agree. Welcome to the ID side! 🙂
Look, Shannon was interested in quantifying bits for transmission and communication purposes. And his theory is not interested in the meaning of the underlying communication. We seem to agree on that point.
With that in mind, can you nevertheless appreciate that meaning of a communication can be of importance, in its own right and irrespective of whatever optimal number of bits gets spit out of a Shannon calculation? Can you accept that ID is interested primarily in the former?
—–
The upshot of all this is that the primary point of my post is underscored: generating random strings that have a high Shannon calculation result do not — and by definition, cannot — invalidate the design inference, because the design inference is not based on simply having a high Shannon calculation number.
Eric (#89):
No, there is no such symbolic connection, as far as I can tell. The string is symbolic, but not the connection.
The only people that I know who would claim (without proof) that the connection is symbolic, are those AI proponents at the extremes of materialism, who claim that humans are just symbol processing computers.
In effect, that amounts to saying that ID is a quest to find an objective account of the subjective. That, too, puts you in agreement with those at the extremes of materialism.
Personally, I see any such quest as doomed to fail.
Neil:
You are confusing two different things. Just because something is created by an intelligence doesn’t mean it is subjective. If I discover a code and can determine what the symbols refer to, then I have learned something real about the code — objectively, just like with mathematics or the genetic code. There is nothing “subjective” about it. If I discover an artifact or a machine and by reverse engineering am able to ascertain its function, there is nothing subjective about that function — it exists, it is real. Don’t make the category mistake of lumping everything that is produced by a mind or by an intelligence into the category of “subjective” (which can then be blithely dismissed without applying intellectual effort to the question at hand).
Furthermore, the attempt to paint the design inference as dealing with things that are “subjective” is no more becoming than the attempt to paint it as dealing with things that are “supernatural.” Both claims are but a rhetorical attempt to dismiss the substance — you know, if it is “supernatural” it isn’t “science”; if it is “subjective” it isn’t objective, and therefore not worth considering.
Not an intellectually worthy approach to take.
Hi Eric,
Sorry, I’ve been occupied with so many other things lately. I haven’t had the time to spend on this that I would like. Perhaps later this week and into next will improve.
But I wanted to get back to you on something.
Eric:
Mung @ 47:
Eric:
For example:
I can come up what more, and will, time permitting.
Meyer is simply confused here (and confusing). By simply arranging ten characters into a string of characters “at random” one does not create information, not even Shannon Information.
And his new ten-digit phone number may not even be a phone number! As Meyer admits.
Neil @ 2:
You’re simply mistaken. The metric is the information. Else it would not be informative, it would not be a metric, it would not be objective, it would be meaningless.
You’re simply mistaken. The metric does not “measure the amount of information.” There is no measure for information.
Eric (#91):
You seem to have missed the point.
I’m saying that meaning is inherently subjective. Whether or not meaning is created by an intelligence, or is just a natural part of organic life, is not relevant to the point that meaning itself is subjective.
If you insist on tying information to meaning, then that makes information subjective. Again, whether or not the information is created by an intelligence is not relevant. It’s the fact that information, as you use that term, is tied to meaning, together with the fact that meaning is subjective.
Mung, thanks for taking time for a few additional comments.
I hope you aren’t quoting me for the notion that:
Shannon information = string of characters.
That was my assessment of Neil’s confused approach @31.
—–
I think you are taking your criticism a bridge too far. If we are saying that “Shannon information” is the resulting number that is spit out when we plug a string into a Shannon calculation, then, yes, Meyer should have been more careful in his light-hearted analogy.
But this brings us back to the very nuanced point: Does a string “contain” Shannon information?
If we are defining Shannon information as the mere numerical result spit out of a calculation, then perhaps no. If we are defining the result spit out of a calculation as a measure of something (say, information carrying capacity), then perhaps yes.
Meyer didn’t “admit” anything on that front. It was part of his point. So you should have said, “As Meyer underscores.”
—–
We seem to be circling around a definitive definition, and I suspect we are very much in agreement.
However, what I would love to hear from you (notwithstanding your wonderful and energetic Socratic method of asking lots of questions), is your definition of Shannon information.
Let’s say I approach you on the street and ask you for a straight-up, unambiguous, no-hidden-nuances, non-rhetorical definition of “Shannon information,” what would you say?
Neil Rickert:
So you’re NOT saying that Shannon Information is meaningless, just that it’s subjective. Thanks for clearing that up. Whew!
Neil Rickert:
If you insist on tying information to meaning, then that makes information subjective.
Make up your mind, please.
Information is inherently tied to meaning. There is no such thing as meaningless information. The very idea is absurd.
Are you asserting that Shannon Information is not tied to meaning? If it isn’t, then what use is it?
Or are you saying Shannon Information is subjective? If so, so what?
Eric:
What was Meyer’s point that he underscores by admitting that the ten digits may not even constitute a phone number? Was it that a random string of characters can provide meaningless information?
Mung, Meyer’s point by underscoring that ten digits may not even constitute a valid phone number is pretty clear. The whole point of those couple of pages is to get people thinking about the difference between mere complexity and specified complexity. His simple example is perfectly suited to that task.
p.107
This is true regardless of whether we define Shannon information as being a property of the string itself or the resulting measurement number of the Shannon calculation.
Again, I’m not sure why you are hung up on this.
More interesting to me though, is how you would define “Shannon information.”
Meyer provides a good starting point for discussing Shannon Information.
Uncertainty, of course, implies uncertainty about something. So Shannon Information in no way reduces the requirement for the aboutness of information. In fact, it is inherent in the very concept of Shannon Information.
The amount of uncertainty about what?
One wonders why Shannon Uncertainty isn’t as popular a concept as Shannon Information. 🙂
To calculate the amount of Shannon Information one must be able to specify the amount of Shannon Uncertainty. And when that is done, it becomes readily apparent what Shannon Information is about. (And what it is not about.
It is a very nuanced point and many people seem to be unaware of the nuance or even of the possibility of the necessity for any nuanced view. The answer is no.
Strings of characters do not “contain” or “carry” or “convey” Shannon Information. Think of Shannon Information as meta-information.
Sorry, I just needed someplace to start and that seemed to be convenient at the time. Didn’t mean to imply that was your PoV. 🙂
Meyer is mistaken, but how (or in how many ways) and why?
Mung, you are a genius at writing riddles. Less so at clear exposition of an answer.
What is Shannon information? Please define.
Eric, see my response at 99.
You don’t agree with Meyer’s definition of Shannon Information or you don’t think he is explaining what Shannon Information is in the quoted material?
RE: 101
How does Meyer know that these characters are from the English alphabet and not from the Latin alphabet?
The fact that symbols may appear to be from the same alphabet doesn’t determine the frequency or probability of their occurrence in a sequence of characters or symbols, and that is what determines how “informative” a given symbol is.
But letters in the English language are not equally probable, so different letters convey different amounts of Shannon Information.
Therefore, both sequences have an equal amount of Shannon information as measured by Shannon’s theory.
The conclusion does not follow. It’s a non sequitur.
There’s no reason to believe both sequences were produced at random. In fact, I think we can be pretty sure the first one was not produced at random.
Perhaps the second sequence came from a source which uses an alphabet identical to that of the first sequence (say one containing 26 symbols plus a space), and perhaps each symbol was equally likely to be transmitted by the source with the same probability (though given the number of spaces I have my doubts), and each character was selected randomly and transmitted.
What the average info per symbol according to Shannon’s theory?
But I find it extremely difficult to believe that the first sequence came from the same source. And why should I?
And if it did not come from a source with the same characteristics, then it doesn’t have “an equal amount of Shannon information as measured by Shannon’s theory.”
: Information: The New Language of Science
: Hans Christian von Baeyer
: Chapter 4
: Counting Bits: The scientific measure of information
: p. 28
In contrast to the vague verbal definition of information, the technical definition, though skeletal, is a model of specificity and succinctness. Claude Shannon, the founder of information theory, invented a way to measure ‘the amount of information’ in a message without defining the word ‘information’ itself, nor even addressing the question of the meaning of the message. He produced, in effect, an operational definition like that of temperature, except that his measuring device – a simple recipe – is not a physical apparatus, like a thermometer, but a conceptual tool.
Shannon’s information measure is most easily applied to a message that consists of a string of binary possibilities – yes or no, heads or tails, zero or one – each of which is equally likely at every step along the string. According to Shannon, each choice corresponds to one bit (short for ‘binary digit’) of information. Communicating to a friend the outcome of three consecutive tosses of a penny, for example, requires three bits of information, which would define eight possible strings of heads and tails. More generally, Shannon’s recipe is simple: To find the information content of any message, translate the message into the binary code of the computer and count the digits of the resulting string of zeros and ones. The number so obtained is called ‘Shannon information’, and the technique is known, somewhat dismissively, as bit-counting.
: Information: The New Language of Science
: Hans christian von Baeyer
: Chapter 12
: Randomness: The flip side of information
: p. 99
Shannon’s technical definition of the information content of a message – the number of digits when the message is written in the binary code of the computer – doesn’t distinguish between sense and nonsense.
Eric, do we have a definition yet of Shannon Information?
If not, why not?
More on the definition of Shannon Information.
Formal-mathematical information A third class of information-theoretic notions includes the formal concepts of information, which have been initially introduced as mathematical tools for measuring the performance of communications devices. The classical notion, in this category, was introduced by the mathematical theory of communication of Shannon (1948) and Shannon and Weaver (1949). In the latter, [Shannon] information is a measure of one’s freedom of choice when one selects a message (the logarithm of the number of available choices or of probabilities).
– Nicolas J. Bullot. Attention, Inforamtion, and Epistemic Perception. In Inforamtion and Living Systems: Philosophical and Scientific Perspectives. George Terzis and Robert Arp, eds.
P1 The Elementary Problem: What Is Information?
Information can be viewed from three perspectives… Many extensionalist approaches to the definition of information as reality or about reality provide different starting points for answering P1.
1. the information theory approach (mathematical theory of codification and communication of data/signals, Shannon and Weaver (1949 rep. 1988) defines [Shannon] information in terms of probability space distribution;
Each extensionalist approach can be given an intentionalist reading, by interpreting the relevant space as a doxastic space, in which information is seen as a reduction in the degree of uncertainty or level of surprise in an informee, given the state of information in the informee.
Information theory in (1) approaches information as a physical phenomenon, syntactically. It is not interested in the usefulness, relevance, meaning, interpretation, or aboutness of data, but in the level of detail and frequency in the uninterpreted data (signals or messages). It provides a successful mathematical theory because its central problem is whether and how much data, not what information is conveyed.
– Luciano Floridi, The Philosophy of Information. p. 30-31
Hey Eric,
In case you ever find time to get back to this thread.
I was reading a book recently that indicated that there are four different Shannon entropies. Would it follow that there are four different measures and that there are thus perhaps four different definitions of “Shannon Information”?
Let’s identify another mistaken ID argument about Shannon Information.
There are many things I find wrong here, but I’ll focus on just one.
Notice the clear connection to the same mistake that Meyer makes.
Shannon’s theory does not and cannot tell us what is required for the creation of information. It does not and cannot ‘allow’ for the creation of ‘information’ by randomly assembling symbols.
Shannon Information is still information. It’s not as if with the appearance of Shannon’s Theorems information gained a little brother, a distinct and separate entity.
However, it is confined to something specific within the greater realm of information: probability distributions.
Shannon Information really is information.
So let’s call it Shannon’s Measure of Information (SMI).
Eric Anderson @ 102:
Your OP is titled “Intelligent Design Basics – Information – Part III – Shannon”
Did you fail to define Shannon Information in the OP? Have you defined Shannon Information anywhere in this thread?
Perhaps we can work together to define Shannon Information.
Assuming, of course, that you are willing to dispense with the claim that Shannon Information is not really information.
What is Shannon’s Measure of Information (SMI) about?
To assert that Shannon information is “not really information” is to assert that there is no “aboutness” to Shannon Information. This assertion ought to be discarded as absurd on it’s face.
“The meaning of information is given by the processes that interpret it.”
– Ed Fredkin
“He [MacKay] proposed that both Shannon and Bavelas were concerned with what he called ‘selective-information,’ that is information calculated by considering the selection of message elements from a set.”
– http://www.physics.utoronto.ca.....h2wii2.pdf
Shannon Information: A Misleading Analogy
From the OP:
Shannon’s Capacity Theorem
See also:
There’s more to Shannon’s paper than just the channel capacity, and it’s this “something more” that people have in mind when they think of Shannon Information.
It’s in Section 6 (Choice, Uncertainty and Entropy) starting on page 392.
So to expand the analogy, Shannon didn’t just compute the carrying capacity of the truck, he developed a way to quantify (measure) the “sizes” (or average size) of the boxes.