A Mathematical Theory of Communication

The problem

The paper begins with a deliberately narrow definition. The fundamental problem of communication is to reproduce at one point, exactly or approximately, a message selected at another point ^[2]. This definition excludes semantics: the semantic aspects of communication are irrelevant to the engineering problem, and what matters is that the actual message is one selected from a set of possible messages ^[2]. That move turns communication into a statistical problem. A source is not treated as a sequence of fixed meanings but as a generator of selections from a set; the engineer can then ask how many possible messages there are, how likely each is, and how faithfully one can be recovered at the destination.

The introduction places the paper against recent modulation methods. PCM and PPM, which exchange bandwidth for signal-to-noise ratio, had intensified interest in a general theory of communication ^[2]. The paper says a basis for such a theory is contained in Nyquist and Hartley, and that it extends that theory to include the effect of noise in the channel, savings possible from the statistical structure of the original message, and the nature of the final destination of the information ^[2]. So the scope is broader than a noiseless channel or a fixed source: noise, source statistics, and destination all enter.

Measurement and capacity

The paper adopts a logarithmic measure of information ^[2]. With base 2, the resulting units can be called binary digits, or more briefly bits, a word suggested by J. W. Tukey ^[2]. A logarithmic scale makes information additive in a natural way when independent choices are combined, and base 2 makes the unit correspond to a binary choice. The paper then defines the capacity C of a discrete channel as the limit, as time grows, of the logarithm of the number of allowed signals of that duration divided by the duration ^[2]. In words, capacity is a rate: how many distinguishable signals the channel can carry per unit time in the long run. It is not a property of one message but of the set of signals the channel permits.

Entropy

The paper also gives a general measure for a set of probabilities. Theorem 2 in Appendix 2 shows that the only function H satisfying the three stated assumptions is H = −K Σ p_i log p_i ^[2]. The paper calls H = −Σ p_i log p_i the entropy of the set of probabilities ^[2]. It notes that the same form appears in statistical mechanics, where H is the H of Boltzmann's H theorem ^[2]. This is a mathematical correspondence, not a claim that communication and thermodynamics are the same subject. Entropy here measures the uncertainty of a probability set; it is the quantity the paper uses to discuss the savings possible from the statistical structure of the original message.

Publication and importance

The paper is by C. E. Shannon and appeared in The Bell System Technical Journal, volume 27, pages 379 to 423 and 623 to 656, in July and October 1948 ^[1]. Its importance lies in the way the parts fit together. A message is a selection from a set ^[2]; information is measured logarithmically, with base 2 giving bits ^[2]; a channel has a capacity defined as a long-run rate ^[2]; and a probability set has an entropy of the form −Σ p_i log p_i ^[2]. These definitions let the paper treat noise, source statistics, and destination within one engineering problem rather than as separate practical difficulties ^[2]. The semantic content is deliberately set aside ^[2], so the theory can be general across kinds of messages. That combination is the paper's contribution: a compact mathematical vocabulary for questions about how much can be sent, how reliably, and how much the structure of the source can save.