Our last chapter is devoted to entropy. With this excuse we first present Shannon’s information theory, including the derivation of his entropy, and the enunciations and proofs of the source coding theorem and of the noisy-channel coding theorem. Then, we consider dynamical systems and the production of entropy in chaotic systems, termed Kolmogorov–Sinai entropy. For non-experts or readers who require a memory jog, we make a short recap of statistical mechanics. That is just enough to tie up some knots left untied in Chapter 4, when we developed large deviations theory for independent variables. Here we generalize to correlated variables and make one application to statistical mechanics. In particular, we find out that entropy is a large deviations function, apart from constants. We end with a lightning fast introduction to configurational entropy in disordered complex systems. Just to give a tiny glimpse of … what we do for a living!

In Chapter 12, we shall examine results for a large class of processes with memory, known as ergodic processes. We start this chapter with a quick review of the main concepts of ergodic theory, then state our main results: Shannon–McMillan theorem, compression limit, and asymptotic equipartition property (AEP). Subsequent sections are dedicated to proofs of the Shannon–McMillan and ergodic theorems. Finally, in the last section we introduce Kolmogorov–Sinai entropy, which associates to a fully deterministic transformation the measure of how “chaotic” it is. This concept plays a very important role in formalizing an apparent paradox: large mechanical systems (such as collections of gas particles) are on the one hand fully deterministic (described by Newton’s laws of motion) and on the other hand have a lot of probabilistic properties (Maxwell distribution of velocities, fluctuations, etc.). Kolmogorov–Sinai entropy shows how these two notions can coexist. In addition it was used to resolve a long-standing open problem in dynamical systems regarding isomorphism of Bernoulli shifts.

In this chapter, we provide a classical account of Kolmogorov–Sinai metric entropy for measure-preserving dynamical systems. We prove the Shannon–McMillan–Breimann Theorem and, based on Abramov's Formula, define the concept of Krengel's Entropy of a conservative system preserving a (possibly infinite) invariant measure.