In many applications, dimensionality reduction is important. Uses of dimensionality reduction include visualization, removing noise, and decreasing compute and memory requirements, such as for image compression. This chapter focuses on low-rank approximation of a matrix. There are theoretical models for why big matrices should be approximately low rank. Low-rank approximations are also used to compress large neural network models to reduce computation and storage. The chapter begins with the classic approach to approximating a matrix by a low-rank matrix, using a nonconvex formulation that has a remarkably simple singular value decomposition solution. It then applies this approach to the source localization application via the multidimensional scaling method and to the photometric stereo application. It then turns to convex formulations of low-rank approximation based on proximal operators that involve singular value shrinkage. It discusses methods for choosing the rank of the approximation, and describes the optimal shrinkage method called OptShrink. It discusses related dimensionality reduction methods including (linear) autoencoders and principal component analysis. It applies the methods to learning low-dimensionality subspaces from training data for subspace-based classification problems. Finally, it extends the method to streaming applications with time-varying data. This chapter bridges the classical singular value decomposition tool with modern applications in signal processing and machine learning.

Here, we define subgradients and subdifferentials of nonsmooth functions. These are a generalization of the concept of gradients for smooth functions, that can be used as the basis of algorithms. We relate subgradients to directional derivatives and to the normal cones associated with convex sets. We introduce composite nonsmooth functions that arise in regularized optimization formulations of data analysis problems and describe optimality conditions for minimizers of these functions. Finally, we describe proximal operators and the Moreau envelope, objects associated with nonsmooth functions that are the basis of algorithms for nonsmooth optimization described in the next chapter.