It is clear from the previous chapter that sample statistics have distributions depending, in general, on the sample size n. The natural question one may ask is how these distributions are affected by varying n deterministically and, in particular, as n increases. It turns out that one can get common limiting results, for a variety of settings, by analyzing what happens as n tends to infinity; this is large-sample asymptotic theory.We study convergence of distributions and of characteristic functions. For variates that converge to a single value, we study convergence in probability and in moments, and almost-sure convergence with its connection to events that happen infinitely-often. The speed of convergence and orders of magnitude are covered, as are law of large numbers (LLNs) weak and strong, limiting distributions such as central limit theorems (CLTs), stable limit theorems (SLTs), the generalized extreme-value theorem with its three special cases, and a law of iterated logarithms.

An extensive list of special univariate distributions is studied, and their main properties analyzed. They are the distributions most often encountered in statistics, and we relate them to one another. Some of these distributions arise out of natural phenomena or have attractive special properties which are explored in the exercises. We also study general classifications of distributions and the ensuing properties, including the exponential family, infinitely divisible distributions, and stable distributions. Distributions contain an inherent amount of information and entropy, which we quantify.

If a vector of variates is transformed into anothervector, what is the resulting distribution? This chapter details the main methods of making such transformations and obtaining the new distribution, density, and characteristic functions. Although the proof of the transformation theorem for densities is not usually given in statistics textbooks, we use the shortcut of conditioning to give a statistical proof. Applications of the three methods range from simple transformation (including convolutions) to products and ratios. General transformations are also studied. These include rotations of vectors (and their Jacobian), transformations from uniform variates to others (useful for generating simulated samples in Monte-Carlo studies) via the probability integral transformation (PIT), exponential tilting, and others. Extreme-value distributions are studied, to complement the study of sample averages. The chapter concludes by revisiting the copula, which transforms the marginals into a joint distribution.

This appendix collects mathematical tools that are needed in the main text. In addition, it gives a brief description of some essential background topics. It is assumed that the reader knows elementary calculus. The topics are grouped in four sections. First, we consider some useful methods of indirect proofs. Second, we introduce basic results for complex numbers and polynomials. The third topic concerns series expansions. Finally, some further calculus is presented, including difference calculus, Stieltjes integrals, and multivariable constrained optimization (covering also the case of inequality constraints).

In the previous chapter, we introduced distributions and density functions as two alternative methods to characterize the randomness of variates. In this chapter, we introduce the final method considered in this book, and relate it to the previous two: the moments of a variate (including mean, variance, and higher-order moments). The condition for the existence of moments is explained and justified mathematically. These moments can be summarized by means of "generating functions". We define moment-generating functions (m.g.f.s). Cumulants and their generating functions are introduced. Characteristic functions (c.f.s) always exist, even if some moments do not, and they identify uniquely the distribution of the variate, so we define c.f.s and the inversion theorem required to transform them into the c.d.f. of a variate. We also study the main inequalities satisfied by moments, such as those resulting from transformations (Jensen) or from comparing moments to probabilities (Markov, Chebyshev). We also show that the mean, median, and mode need not be linked by inequalities, as previously thought.