Information theory

What it is

Information theory is the study of information as a quantity — something that can be measured, encoded and transmitted — rather than as meaning. Its founding document is Shannon's 1948 paper "A Mathematical Theory of Communication", published in the Bell System Technical Journal.^[1] The problem that paper sets itself is deliberately narrow: reproducing at one point, either exactly or approximately, a message selected at another point.^[1]

That framing has a striking consequence. The semantic aspects of a message are irrelevant to the engineering problem, because what matters is that the message is drawn from a set of possible messages.^[1] If the receiver already knows which message will arrive, nothing needs to be sent; the information lies in the uncertainty about which member of the set was chosen. Information theory therefore begins by putting meaning aside and asking only how many alternatives a source offers, and how faithfully a channel can carry a choice among them.

How it works

Information is measured on a logarithmic scale, and the base-2 unit is called the bit, a term suggested by J. W. Tukey.^[1] A logarithmic measure has the useful property that independent choices add rather than multiply: the amount of information in two separate decisions is the sum of the amounts in each.

The central quantity is entropy, written H and defined as minus the sum, over the possible outcomes, of each probability multiplied by the base-2 logarithm of that probability.^[1] The paper uses it as a measure of information, choice and uncertainty.^[1] A small worked case makes it concrete: a source with two equally likely outcomes has an entropy of 1 bit, one with four equally likely outcomes has 2 bits, and an outcome that is already certain contributes nothing at all.

Statistical structure in a message is a resource rather than an obstacle. The paper shows that regularities such as the differing frequencies of letters in English can be exploited by encoding to save time or channel capacity.^[1] A code that gives short representations to common letters and long ones to rare letters spends fewer bits on average than one that treats every letter alike, and the saving comes precisely from structure that a naive scheme ignores.

For a channel, the paper defines a capacity C through the way the number of allowed signals grows with their duration.^[1] It also extends the theory to take account of noise inside the channel, instead of treating noise as an external nuisance to be handled separately.^[1] Source, code and channel are thus described in one vocabulary.

Where it came from

The theory arrived as a single paper in 1948 and became the founding document of the field.^[1] Its originality lies not in a new device but in a new set of definitions: a unit for information, a measure of a source's uncertainty, and a property of a channel. Once these were named, statements about communication could be made in numbers.

Where it is used

Wherever a message has to be carried from one place to another — along a wire, over a radio link, onto a storage medium — the questions the paper separates are the questions that have to be answered: how much uncertainty a source produces, how much a channel can carry, and what an encoding costs.^[1] The decisive move is the separation of the measurement of information from meaning; only because the two are separated does information become something an engineer can compute with.^[1] The same definitions also give a common language for comparing schemes that look very different on the surface, since each can be described by the statistics of its messages and the capacity of its channel.^[1]

Limits

The theory sets meaning aside on purpose, so questions about whether a message is true, useful or beautiful fall outside it.^[1] Its savings depend on there being statistical structure in the message to exploit.^[1] And noise is handled by building it into the model of the channel rather than by pretending it is absent.^[1] Within those boundaries the framework is exact; outside them it says nothing.