Chapter 1 introduces the first information measure – Shannon entropy. After studying its standard properties (chain rule, conditioning), we will briefly describe how one could arrive at its definition. We discuss axiomatic characterization, the historical development in statistical mechanics, as well as the underlying combinatorial foundation (“method of types”). We close the chapter with Han’s and Shearer’s inequalities, which both exploit the submodularity of entropy.

Any deterministic process loses information, and one can quantify the amount of information lost. Information loss is a generalization of entropy, and in some ways is a better-behaved quantity, being more functorial. We give a simple axiomatic characterization of information loss.

We give a short introduction to some classical information-theoretic quantities: joint entropy, conditional entropy and mutual information. We then interpret their exponentials ecologically, as meaningful measures of subcommunities of a larger metacommunity. These subcommunity and metacommunity measures have excellent logical properties, as we establish. We also show how all these quantities can be presented in terms of relative entropy and the value measures of the previous chapter.