Throwing Away Meaning

Claude Shannon's 1948 paper opens by setting aside the thing everybody assumes information is about.

The semantic aspects of communication, he writes, are irrelevant to the engineering problem. Whether a message means anything, whether it is true, whether anybody cares: none of it matters for the question of getting it from one place to another.

Discarding meaning is what made the subject possible, and it is why the resulting mathematics applies equally to speech, text, images, DNA and neural spike trains.

What is left, once meaning is gone, is surprise. A message that tells you something you already expected carries little information. A message you could not have predicted carries a lot. Shannon made this precise: the information content of an event is the logarithm of one over its probability, measured in bits.

A coin flip carries one bit, because it halves the space of possibilities. A message that was certain carries zero bits, because you already knew.

Entropy is the average information content of a source, and it sets a hard floor on compression. A source emitting the same symbol forever has zero entropy and needs no bits. A source emitting random symbols has maximum entropy and cannot be compressed at all. Shannon proved you cannot compress below a source's entropy without losing information.

English text has an entropy of roughly one bit per letter, against the 4.7 bits a naive encoding of 26 letters would need, which means ordinary written English is around 75% redundant. That is why you can rd ths sntnce, and why text compresses so well.

It also means one particular claim is provably false, and it recurs often enough to be worth being able to dismiss on sight. A compressor that shrinks every possible input cannot exist: there are fewer short strings than long ones, so any scheme mapping every input to a shorter output must send two distinct inputs to the same output and cannot reverse them. Every real compressor shrinks structured data and slightly expands random data, which is fine, because real files have structure. Investors have occasionally funded companies claiming otherwise.

The result that genuinely astonished contemporaries concerns noise.

Everyone assumed reliability and speed traded off: to get a message through a noisy channel you must slow down, repeat yourself, and even then some errors get through. More noise means worse communication, continuously.

Shannon proved that every channel has a capacity, and that below that rate you can achieve an arbitrarily low error rate (not a low one, arbitrarily low) while above it you cannot do better than a fixed error however clever you are. There is a cliff rather than a slope.

He proved codes achieving this exist without constructing one, using a probabilistic argument showing that a randomly chosen code would work. Actually finding practical codes that approach the limit took nearly fifty years, and turbo codes and LDPC codes now come within a fraction of a decibel of it.

Which turns out to depend on redundancy being added rather than removed. A well-designed error-correcting code adds structured redundancy that lets a receiver detect and repair corruption without asking for a retransmission, and over a noisy channel that is dramatically more efficient than sending a compact message that arrives wrong.

Every storage medium, every mobile network and every deep-space probe runs on this. The Voyager spacecraft are still returning data from beyond the solar system, at power levels comparable to a refrigerator light bulb across billions of kilometres, because their error-correcting codes let a receiver reconstruct the message from a signal barely distinguishable from noise.

And the abstraction from meaning is what lets the framework travel.

DNA is a channel transmitting a message across generations, with a measurable error rate and a repair mechanism, and the redundancy of the genetic code, several codons specifying the same amino acid, is error correction. A neuron's spike train has an information rate that can be measured in bits per second. A scientific experiment reduces uncertainty by a quantifiable amount, which is the basis of information-theoretic approaches to experimental design.

None of these are metaphors. They are applications of the same theorems, available because Shannon declined to talk about what messages mean.