This chapter covers principal component analysis and low-rank models, which are popular techniques to process high-dimensional datasets with many features. We begin by defining the mean of random vectors and random matrices. Then, we introduce the covariance matrix which encodes the variance of any linear combination of the entries in a random vector, and explain how to estimate it from data. We model the geographic location of Canadian cities as a running example. Next, we present principal component analysis (PCA), a method to extract the directions of maximum variance in a dataset. We explain how to use PCA to find optimal low-dimensional representations of high-dimensional data and apply it to a dataset of human faces. Then, we introduce low-rank models for matrix-valued data and describe how to fit them using the singular-value decomposition. We show that this approach is able to automatically identify meaningful patterns in real-world weather data. Finally, we explain how to estimate missing entries in a matrix under a low-rank assumption and apply this methodology to predict movie ratings via collaborative filtering.

This chapter introduces matrix factorizations – somewhat like the reverse of matrix multiplication. It starts with the eigendecomposition of symmetric matrices, then generalizes to normal and asymmetric matrices. It introduces the basics of the singular value decomposition (SVD) of general matrices. It discusses a simple application of the SVD that uses the largest singular value of a matrix (the spectral norm), posed as an optimization problem, and then describes optimization problems related to eigenvalues and the smallest singular value. (The “real” SVD applications appear in subsequent chapters.) It discusses the special situations when one can relate the eigendecomposition and an SVD of a matrix, leading to the special class of positive (semi)definite matrices. Along the way there are quite a few small eigendecomposition and SVD examples.

In this chapter, we develop spectral techniques. We highlight some applications to Markov chain mixing and network analysis. The main tools are the spectral theorem and the variational characterization of eigenvalues, which we review together with some related results. We also give a brief introduction to spectral graph theory and detail an application to community recovery. Then we apply the spectral theorem to reversible Markov chains. In particular we define the spectral gap and establish its close relationship to the mixing time. We also show in that the spectral gap can be bounded using certain isoperimetric properties of the underlying network. We prove Cheeger’s inequality, which quantifies this relationship, and introduce expander graphs, an important family of graphs with good “expansion.” Applications to mixing times are also discussed. One specific technique is the “canonical paths method,” which bounds the spectral graph by formalizing a notion of congestion in the network.

Chapter 14: Every square complex matrix is unitarily similar to an upper triangular matrix, but which matrices are unitarily similar to a diagonal matrix? The answer is the main result of this chapter: the spectral theorem for normal matrices. Hermitian, skew-Hermitian, unitary, and circulant matrices are unitarily diagonalizable. As a consequence, they have special properties that we investigate in this and following chapters.

We provide a brief, but self-contained, introduction to the theory of self-adjoint operators. In a first section we give the relevant definitions, including that of the spectrum of a self-adjoint operator, and we discuss the proof of the spectral theorem. In a second section, we discuss the connection between lower semibounded, self-adjoint operators and lower semibounded, closed quadratic forms, and we derive the variational characterization of eigenvalues in the form of Glazman’s lemma and of the Courant–Fischer–Weyl min-max principle. Furthermore, we discuss continuity properties of Riesz means and present in abstract form the Birman–Schwinger principle.

This chapter gives an introduction to orthogonal polynomials. It also includes the concept of the Stieltjes transform and some of its properties, which will play a very important role in the spectral analysis of discrete-time birth–death chains and birth–death processes. A section on the spectral theorem for orthogonal polynomials (or Favard’s theorem) will give insights into the relationship between tridiagonal Jacobi matrices and spectral probability measures. The chapter then focuses then on the classical families of orthogonal polynomials, of both continuous and discrete variables. These families are characterized as eigenfunctions of second-order differentials or difference operators of hypergeometric type solving certain Sturm–Liouville problems. These classical families are part of the so-called Askey scheme.

This appendix feeds into Appendix C by developing the properties of a particular symmetric operator on a Hilbert space defined by an integral, by showing it is compact and giving a proof ofthe spectral theorem.

Many special functions are eigenfunctions to explicit operators, such as difference and differential operators, which is in particular true for the special functions occurring in the Askey scheme, its q-analogue and extensions. The study of the spectral properties of such operators leads to explicit information for the corresponding special functions. We discuss several instances of this application, involving orthogonal polynomials and their matrix-valued analogues.