If science represents humanity's endeavor to understand the universe, then probability theory is the language in which we encode this knowledge. In essence, probability theory formalizes the effect that evidence has on what we believe. Similarly, information theory quantitatively summarizes our knowledge. In this appendix, I briefly describe how probability theory and information theory developed, and how they serve to evaluate scientific hypotheses.

Probability theory

Basic concepts

This section briefly describes the central concepts required to understand the remainder of this appendix. For a more rigorous, detailed presentation of these concepts, please refer to.

Without loss of generality, we consider in this section only binary (i.e., two-valued) variables. Most of the concepts described in this section generalize to multi-state variables and continuous variables. The marginal probability that a statement S is true, written as P(S), is the probability that statement S is true in the absence of any evidence. The joint probability of statements S1 and S2, written as P(S1, S2), is the probability that both statements are true simultaneously, again in the absence of any evidence. Mutually exclusive statements cannot be true simultaneously; for example, the statements “the patient has squamous-cell carcinoma of the lung” and “the patient does not have squamous-cell carcinoma of the lung” are mutually exclusive.