The fourth chapter introduces the singular value decomposition (SVD), a fundamental matrix factorization with broad applications in data science. The chapter begins by reviewing key linear algebra concepts, including matrix rank and the spectral theorem. It then explores the problem of finding the best low-dimensional approximating subspace to a set of data points, leading to the formal definition of the SVD. The power iteration method is presented as an efficient way to compute the top singular vectors and values. The chapter then demonstrates the application of SVD to principal components analysis (PCA), a dimensionality reduction technique that identifies the directions of maximum variance in data. Further applications of the SVD are discussed, including low-rank matrix approximations and ridge regression, a regularization technique for handling multicollinearity in linear systems.

The correlation coefficient measures the linear relation between scalar X and scalar Y. How can the linear relation between vector X and vector Y be measured?Canonical Correlation Analysis (CCA) provides a way. CCA finds a linear combination of X, and a (separate) linear combination of Y, that maximizes the correlation. The resulting maximized correlation is called a canonical correlation. More generally, CCA decomposes two sets of variables into an ordered sequence of component pairs ordered such that the first pair has maximum correlation, the second has maximum correlation subject to being uncorrelated with the first, and so on. The entire decomposition can be derived from a Singular Value Decomposition of a suitable matrix. If the dimension of the X and Y vectors is too large, overfitting becomes a problem. In this case, CCA often is computed using a few principal components of X and Y. The criterion for selecting the number of principal components is not standard. The Mutual Information Criterion (MIC) introduced in Chapter 14 is used in this chapter.