This chapter discusses techniques to build predictive models from data and to quantify the uncertainty of the model parameters and of the model predictions. The chapter discusses important concepts of linear and nonlinear regression and focuses on a couple of major paradigms used for estimation: maximum likelihood and Bayesian estimation. The chapter also discusses how to incorporate prior knowledge in the estimation process.

This chapter provides an end-to-end introduction to statistics; this highlights how statistics can be used to develop models from data, to quantify the uncertainty of such models, and to make decisions under uncertainty. The chapter also discusses how random variables are the key modeling paradigm that is used in statistics to characterize and quantify uncertainty and risk.

This chapter provides a discussion on multivariate random variables, which are collections of univariate random variables. The chapter discusses how the presence of multiple random variables gives rise to concepts of covariance and correlation, which capture relationships that can arise between variables. The chapter also discussed the multivariate Gaussian model, which is widely used in applications.

This chapter discusses how to apply principles of statistics, optimization, and linear algebra in advanced techniques of data science and machine learning. The chapter shows how to use principal component analysis and singular value decomposition for analyzing complex datasets and discusses advanced estimation techniques such as logistic regression, Gaussian process models, and neural networks.

This chapter provides an overview of different theoretical random variable models that can be used to model random phenomena encountered in applications. The chapter discusses the types of behavior that different models capture and provides some preliminary discussion on how to determine model parameters from data.

This chapter discusses techniques that help us estimate parameters and summarizing statistics for random variables from data. The chapter discusses techniques such as the method of moments, least-squares, and maximum likelihood. The chapter also touches on concepts of Monte Carlo simulation, which is a technique that can be used to approximate the summarizing statistics of random variables from random samples or from data. The chapter also highlights how one can characterize the quality of such approximations using the central limit theorem and the law of large numbers.

This chapter discusses techniques to measure uncertainty/risk and to make decisions that explicitly take risk into consideration. The chapter also discusses how to use principles of statistics and optimization in advanced decision-making techniques such as stochastic programming, flexibility analysis, and Bayesian optimization.

The objective of this chapter is to extend the ad hoc least squares method of somewhat arbitrarily selected base functions to a more generic method applicable to a broad range of functions – the Fourier series, which is an expansion of a relatively arbitrary function (with certain smoothness requirement and finite jumps at worst) with a series of sinusoidal functions. An important mathematical reason for using Fourier series is its “completeness” and almost guaranteed convergence. Here “completeness” means that the error goes to zero when the whole Fourier series with infinite base function is used. In other words, the Fourier series formed by the selected sinusoidal functions is sufficient to linearly combine into a function that converges to an arbitrary continuous function. This chapter on Fourier series will lay out a foundation that will lead to Fourier Transform and spectrum analysis. In this sense, this chapter is important as it provides background information and theoretical preparation.

The objective of this chapter is to present some important relations between the Fourier Transform and correlation functions. It turns out that the cross-correlation function and autocorrelation function have some useful relationships to Fourier Transform and power spectrum of the individual functions. As a result, cospectrum and coherence (normalized statistical correlation in frequency domain) can be defined.