An Outline of the Element

This Element contains two parts: Foundations and Applications. The Foundations part contains Sections 2, 3 and 4, in which we introduce the fundamentals of financial time-series and deep learning algorithms. The Applications contains Sections 5, 6, and 7, in which we discuss prediction, portfolio optimization, trade execution and real-world applications.

• Section 2 discusses the statistics frequently used in the analysis of financial time-series, including returns, data distributions, hypothesis testing, statistical moments, serial covariance, correlation, and statistical time-series models such as AR and ARMA. This section also introduces the notions of “alpha” and “beta” and examines the phenomenon of volatility clustering.

• Section 3 introduces supervised learning and its primary components, including loss functions and evaluation metrics. We then introduce neural networks, starting with the canonical fully connected layers, convolutional and recurrent layers. Finally, we explore some state-of-the-art networks, including WaveNet, encoder-decoders, and transformers.

• Section 4 presents a complete training workflow from the very first step of data collection through the final model deployment. We discuss the problem of overfitting and introduce cross-validation for hyperparameter tuning. We also include a discussion of various popular model pipelines so that readers can choose the most appropriate platform for their respective applications.

• Section 5 introduces classical quantitative strategies such as time-series momentum and cross-sectional momentum strategies, and shows how they can be enhanced with deep learning methods. In particular, we explore networks that directly output trade positions and are end-to-end optimized for Sharpe ratio or other performance metrics.

• Section 6 focuses on risk management and portfolio optimization. We demonstrate how deep learning models can help better forecast risk measures such as volatility. We also look into end-to-end deep learning frameworks for portfolio optimization, bypassing the need to estimate returns or construct a covariance matrix for classical mean-variance problems.

• Section 7 introduces high-frequency microstructure data. We demonstrate how bespoke hybrid-networks can serve to forecast future price trends and exploit additional structure in limit order books. Additionally, we discuss various promising applications including the adoption of reinforcement learning for trade execution and generative modeling for financial data.

• Section 8 brings together the insights and knowledge presented throughout this Element, summarizing the key takeaways from our exploration of deep learning and quantitative trading. Looking ahead, we discuss emerging trends and explore future possibilities where deep learning might bring innovative transformations to financial markets.

2.1 Returns

Returns are a key metric in the field of finance, playing an important role in evaluating investment performance over time. They reflect the profit or loss achieved relative to the initial value of an investment, demonstrating insights into the potential profitability and risks associated with different traded financial assets, including stocks, bonds, mutual funds, and other instruments. By calculating and analyzing returns, investors, analysts, and portfolio managers can assess the effectiveness of their strategies, compare diverse investment options, and make data-driven decisions to enhance trading performance.

There are several different ways to calculate financial returns, with simple returns and logarithmic (log) returns being the most common. Simple returns calculate the percentage change in an asset’s price between two consecutive periods. They are straightforward to calculate and understand, which makes them a widely used tool for routine financial analysis. In addition, simple returns are often used to calculate portfolio returns (which will be defined in later sections). However, simple returns have limitations, particularly when dealing with long-term investments or compounding returns.

Logarithmic returns, on the other hand, calculate the natural logarithm of the ratio of consecutive prices. This method provides a time-additive measure, meaning that the returns over multiple periods can be summed up to obtain the total return, which is particularly useful for continuous compounding contexts. Logarithmic returns are often more statistically desirable due to their properties, such as normality and symmetry, which make them more suitable for sophisticated financial models and risk assessments. The motivation for using those two forms becomes apparent when calculating the cumulative return of a security. For a single time step, we can define both returns as follows:

$\begin{array}{c}\text{Simple return: }{r}_{sim,t}=\frac{p_t-p_{t-1}}{p_{t-1}},\\ \text{Log return: }{r}_{log,t}=\log \left(\frac{p_t}{p_{t-1}}\right),\end{array}$(1)

where $p_t$ denotes the price of a security at time $t$. The aforementioned can easily be generalized for returns over multiple time steps from $t-L$ to $t$.

Understanding and analyzing financial returns is crucial for several reasons. First, returns directly impact an investor’s wealth and financial planning, as they determine the growth of investments over time. Second, returns are used when assessing investment risks, and effective risk control is the key to ensuring long-term investment success. Third, analyzing historical returns helps investors identify trends and patterns, informing future investment decisions and strategy development. Finally, financial institutions and fund managers rely heavily upon return analysis to manage large portfolios and ensure they meet their performance benchmarks. By examining returns, they can allocate assets more effectively, diversify their portfolios, and carry out risk management strategies that can protect profits against adverse market movements.

In summary, financial returns are a cornerstone of investment analysis and decision-making. They provide a complete view of the performance and risk of financial assets, guiding investors and financial professionals in their pursuit of optimal investment strategies and wealth maximization. Understanding the different methods of calculating returns is important for anyone involved in the financial world. In many of the data-driven examples that we cover in this Element, a future return over a specific horizon serves as the target of a predictive supervised learning model. It reflects the direction and extent of the expected future price movement and plays a major role in portfolio optimization, which will be discussed more in later sections.

2.2 Distributions of Financial Returns

Loosely speaking, a distribution describes the way in which values of a random variable are spread or dispersed. Distributions are the foundation for the domains of probability and statistics, and distributions can be either discrete, where data points can take on values from a finite or a countable set, or continuous, where data points can take on any value within a given range. Understanding distributions is useful for making inferences about populations based on samples, assessing probabilities, and conducting various statistical tests.

Mathematically, we represent the distribution of a discrete variable by a probability mass function (PMF) and that of a continuous variable by a probability density function (PDF). The PMF simply indicates the probabilities of different finite or countable outcomes, and the PDF presents how the probability of a random variable taking values in a specific range is distributed. The key properties of PMFs or PDFs are that they are nonnegative and sum to one over the entire space of possible values. Taking a continuous variable $X$ with the PDF $f(x)$ as an example, the probability that $X$ lies within the interval $[a,b]$ is determined by the integral of $f(x)$ over that range:

$P(a\le X\le b)={\int }_a^{b}f(x)dx.$(2)

It is important to understand the concept of the distribution of financial returns as it gauges the quality and risk of investment performance. In practice, financial returns do not typically follow a normal distribution, which would suggest they tend not to be well behaved. Instead, they exhibit characteristics, such as heavy tails which indicate that extreme values (large gains or losses) are more frequent than would be predicted by a normal distribution.

There are several ways to understand a distribution. The most straightforward way is to use histograms to visually inspect the data distribution. For example, the left plot of Figure 1 depicts the histogram for the simple daily returns of Standard & Poor’s 500 (S&P500) since its creation. The distribution is bell-shaped and appears similar to a normal (Gaussian) distribution. However, upon inspection, this distribution exhibits fatter tails and a sharper peak compared to a normal distribution with the same mean and standard deviation (plotted in black).Footnote ³

Figure 1 Left: histogram for the return distribution; Right: QQ-plot.

Figure 1Long description

In the left graph, the x-axis ranges from minus 0.20 to 0.10, and the y-axis ranges from 0 to 60 with increments of 10. The data are follow: (minus 0.05, 0), (0.00, 40), (0.05, 0). In the right graph, the x-axis represents Theoretical Quantiles ranging from minus 4 to 4 and the y-axis represents Ordered values ranging from minus 0.20 to 0.10. A line drawn through the points (minus 4, minus 0.05), (0, 0.00), (4, 0.04). All values are approximate.

QQ-plot (quantile-to-quantile plot) is another popular tool to check if a data distribution is normal. A QQ-plot is made by plotting one set of quantiles against another set of quantiles. A quantile denotes an input value, in this case a return, such that a certain fraction (say 90%) of the data are less than or equal to this value (in which case we call it the 90% quantile). A straight line would be expected if two sets of quantiles are from the same distribution. The right of Figure 1 is the QQ-plot for the return distribution of the S&P 500 versus the assumed normal distribution. We can observe that the points align along a straight line in the central portion of the figure, but curve off at the two ends. In general, a QQ-plot like this indicates that extreme values are more likely to occur than the assumed normal distribution. Code to create a histogram and a QQ-plot in Python is shown next:

Code 1.1

Code 1.1

1 
2 
3 
4 
5
6 
7 
8 
9 
10 
11 
12 
13
14 
15 
16 
17 

Empirical studies of financial markets have shown that the distribution of returns is often better fit by distributions such as the Student’s t-distribution, which better accounts for the heavy tails, or the Generalized Error Distribution (GED). These distributions offer a more accurate depiction of the probability of extreme events, and consequently better model financial time-series. We need to be aware of such properties because a higher likelihood of extreme events means that financial models have a higher potential for significant losses.

Overall, a good understanding of return distributions helps with financial modeling, risk assessment, and strategic decision-making. By recognizing that returns are not normally distributed and accounting for the actual distributional characteristics, investors and analysts can develop more robust models that better capture the risks and potential rewards of their investments. This knowledge allows for more effective portfolio diversification, hedging strategies, and overall risk management practices, ultimately leading to more informed and potentially profitable investment decisions.

2.3 Statistical Moments

Statistical moments are sets of parameters used to describe a distribution. In general, we can define the $k$-th central moment of a random variable $X$ as:

${\mu }_k=E[(X-\mu {)}^k],\mu =E(X),$(3)

for $k\ge 2$ and ${\mu }_1=\mu$. Typically, attention is given to the first four moments – mean (or expected value), variance, skewness, and kurtosis – as they capture a distribution’s central tendency, spread, asymmetry, and peakedness, respectively. Statistical moments help us understand the behavior of a distribution and make predictions. For example, a normal distribution can be specified by giving a mean and standard deviation. However, these two moments alone might not be enough to fully describe a return distribution. As an example, the return distribution in Figure 1 exhibits heavier tails and a more pronounced peak than a normal distribution. In this case, we need to check higher moments of the distribution to better understand the data.

In statistics, skewness and kurtosis are the normalized third and fourth central moments. Skewness measures the asymmetry of data about its mean. There are two types of skewness: positive skew and negative skew. A symmetrical distribution, such as a normal distribution, has no skewness. However, a distribution that has larger values on the right tail is positively skewed. On the contrary, a negatively skewed distribution has larger values in the left tail that are further from the mean than those of the left tail.

In finance, skewness may stem from diverse market forces. Investor sentiment can lead to asymmetrical buying or selling pressures as market participants overreact to news or trends. Economic news can also introduce sudden, unidirectional shocks to asset prices as markets rapidly adjust to new information. Market microstructure might also contribute to skewness when imbalances in order flow, liquidity constraints, or trading mechanisms create price distortions. There are many other possible causes for deviations from a normal distribution in the returns.

The skewness of a return distribution can typically inform the reward profile of a security or strategy. A canonical example of a strategy that is negatively skewed is a reversion strategy. We can expect many small positive rewards when assets revert as expected, but we can also suffer large losses if reversion does not occur say due to an unexpected news event. Selling options and VIX futures are other examples of strategies with negatively skewed return distributions. Vice versa, a positively skewed return distribution typically corresponds to many small losses with a few large gains – a canonical example being momentum strategies. The most favorable type of skewness depends upon the risk preferences of investors.

Kurtosis is the fourth normalized statistical moment, describing the tail and peak of a distribution. In particular, kurtosis informs us whether a distribution includes more extreme values than a normal distribution. All normal distributions, regardless of mean and variance, have a kurtosis of 3. If a distribution is highly peaked and has fat tails, its kurtosis is greater than 3, and, vice versa, a flatter distribution has a kurtosis lesser than 3. Excess kurtosis can be attributed to market shocks, economic crises, and other rare but impactful events that significantly affect asset prices.

2.4 Statistical Hypothesis Testing

Hypothesis testing is another important concept in statistics, as it offers a systematic framework for making decisions and drawing conclusions about a population using sample data. It is widely adopted in several fields, including natural science, economics, psychology, and finance, where it is used to evaluate hypotheses and determine the validity of claims or theories. For example, we have already mentioned the fact that financial returns can have fat tails compared to a normal distribution. To objectively assess this, we resort to hypothesis testing which provides a formal framework for making inferences and offers a structured method for evaluating claims. Consequently, this allows researchers and analysts to draw conclusions with a quantifiable level of confidence. By using statistical techniques and predefined criteria, it also eliminates subjective biases and ensures that decisions are conditioned on empirical evidence rather than intuition or guesswork.

The fundamental concept of hypothesis testing involves using sample data to evaluate the evidence against a null hypothesis ($H_0$), which functions as the default or baseline assumption. The objective is to prove the alternative hypothesis ($H_1$), which constitutes the presence of a difference from the default assumption. The process of hypothesis testing involves several key steps. First, we need to define the null hypothesis and the alternative hypothesis. For instance, in a test to determine whether the distribution of returns is normal, the null hypothesis might state that the return distribution follows a normal distribution, while the alternative hypothesis might posit that the return distribution violates the assumption of normality.

We then need to select a significance level ($\alpha$), typically 0.05, which delineates the probability of erroneously rejecting the null hypothesis when it is in fact true. The value of $\alpha$ reflects the strength of the evidence for rejection, and thus a small $\alpha$ imposes a requirement of strong evidence for the null hypothesis to be rejected. In addition, it is necessary to select an appropriate test statistic in accordance with the nature of the data and the hypothesis under examination. Popular test statistics include the z-score, t-score, F-statistic, and chi-square statistic.Footnote ⁴ Following is a complete example of one-sample hypothesis testing to determine whether the mean of a population is zero. Suppose we have a random sample $X_1,X_2,\cdots ,X_n$ from some unknown distribution. We assume that the sample mean is approximately normally distributed for large $n$, and we want to test the null and alternative hypotheses:

$\begin{array}{rl}{H}_0:\mu & =0,\\ H_1:\mu & \ne 0,\end{array}$(4)

where $\mu$ is the true mean of the population. To calculate the test statistic, we use a t-statistic with $n-1$ degrees of freedom:

$T=\frac{\bar{X}-\mu }{s/\sqrt{n}}=\frac{\bar{X}}{s/\sqrt{n}},$(5)

where $\bar{X}$ and $s$ are the sample mean and standard deviation. Under $H_0$, this statistic approximately follows a t-distribution with $n-1$ degrees of freedom.

After computing the test statistic using sample data, we derive a value that can be compared against a critical value for a given alpha. The probability of observing a test statistic, under $H_0$, that is more extreme than the one we computed from our data is called the $p$-value. The $p$-value decides the statistical significance of our results compared to the null hypothesis. In the previous example, we can find $p=2\times P(T_{n-1}>|t_{obs}|)$ where $t_{obs}$ is the observed value of the statistic, and $T_{n-1}$ denotes a random variable following the t-distribution with $n-1$ degrees of freedom. If the resulting $p$-value is smaller than the significance level, for example, $\alpha =0.05$, then we can say the test result is significant, indicating strong evidence against the null hypothesis.

In the previous section, we use graphical tools to assess data distributions but we can now also use statistical hypothesis testing to validate data properties. For instance, the Jarque-Bera test can be utilized to assess the validity of the normality assumption. This widely recognized statistical method evaluates whether the sample data skewness and kurtosis are consistent with those expected in a normal distribution, thereby determining if the return distribution adheres to normality.

2.5 Serial Covariance, Correlation, and Stationarity

For time-series forecasting, we make predictions based on historical observations from previous time stamps. While increments in financial time-series are close to independent random variables, and are indeed often modeled by stochastic processes such as Geometric Brownian Motion (GBM), it is still possible to identify and exploit small dependencies to make predictions. We can measure such dependence with serial covariances (autocovariances) or serial correlations (autocorrelations).

Serial covariance refers to the measure of how two variables change together over time within a time-series. In the context of financial time-series, it specifically measures the covariance between different observations of the same financial variable at different points in time. Intuitively, autocovariance tells us how two instances of a time-series $(x_t,x_s)$ at different time points move together. Understanding serial covariance is beneficial for identifying patterns and predicting future values based on historical data. For example, if the price of a stock today is positively correlated with its price yesterday, this might indicate a future upward trend.

Correlation quantifies both the magnitude and the orientation of the linear association between two time-series. In contrast to covariance, correlation is both standardized and dimensionless, offering a uniform metric for assessing the extent to which two variables vary together. The correlation coefficient spans from -1 to 1, where 1 signifies a flawless positive linear association, -1 denotes a perfect negative linear relationship, and 0 implies the absence of any linear relationship. Formally, we can define covariance and correlation as:

$\begin{array}{rl}\text{Covariance: }Cov(X_s,X_t)& =E[(X_s-E(X_s))(X_t-E(X_t))],\\ & =E(X_s{X}_t)-E(X_s)E(X_t),\\ \text{Correlation: }Corr(X_s,X_t)& =CoV(X_s,X_t)/\sqrt{Var(X_s)Var(X_t)}.\end{array}$(6)

Correlation is essential for many applications in financial time-series analysis. By evaluating the extent to which two assets fluctuate together, investors are able to design portfolios that reduce risk while optimizing returns. For instance, when two assets exhibit low or negative correlations, a portfolio combining the two can achieve lower overall volatility compared to portfolios consisting of highly correlated assets. Correlation analysis is also used for detecting market inefficiencies and arbitrage opportunities. For example, if two assets are expected to be highly correlated due to economic or financial reasons but deviate significantly at some point, one could speculate that this deviation will shrink again.

We can check how a time-series relates to previous observations (at various time lags) using Autocorrelation Function (ACF) plots and Partial Autocorrelation Function (PACF) plots. The ACF measures the correlation between a time-series and its own lagged (i.e., past) values. It indicates the degree to which past values of a series influence its current values, providing insights into the internal structure and patterns of the data. The ACF is especially effective in detecting trends, seasonal patterns, and various cyclical behaviors within a dataset. Mathematically, the ACF at lag $k$ for a time-series $X_t$ is defined as:

${\rho }_k=Corr(X_t,X_{t+k})=\frac{Cov(X_t,X_{t+k})}{{\sigma }^2},$(7)

where $Cov(X_t,X_{t+k})$ is the covariance between $X_t$ and $X_{t+k}$, and ${\sigma }^2$ is the variance of the time-series. The values of the ACF span from -1 to 1, where values approaching 1 signify a robust positive correlation, those nearing -1 indicate a strong negative correlation, and values around 0 imply minimal to no correlation. A correlogram depicts the autocorrelation of a time-series as a function of time lags. This plot can help identify the appropriate model for time-series forecasting, such as an Autoregressive Moving Average model (ARMA), in which the autocorrelation structure guides the selection of model parameters.

The PACF serves as an additional instrument in time-series analysis, quantifying the correlation between a time-series and its lagged values while also eliminating the linear effects of intermediate lags. Unlike the ACF, which includes the cumulative effect of all previous lags, the PACF isolates the direct effect of a specific lag. For instance, the PACF at lag $k$ measures the correlation between $X_t$ and $X_{t+k}$ after removing the effects of lags 1 through $k-1$. This allows for a clearer understanding of the underlying relationship at each specific lag, making it easier to identify the appropriate number of lags to include in an autoregressive model.

Mathematically, we can define the PACF at lag $k$ as the correlation between $X_t$ and $X_{t+k}$ that is not accounted for by their mutual correlation with $X_{t+1},X_{t+2},$ $\cdots ,X_{t+k-1}$. We can obtain PACF values by fitting a linear model with $X_t$ and the regressors standardized:

$X_t=\alpha +{\varphi }_{k,1}{X}_{t-1}+{\varphi }_{k,2}{X}_{t-2}+\cdots +{\varphi }_{k,k}{X}_{t-k},$(8)

where ${\varphi }_{k,k}$ is the PACF value for lag $k$, and ranges from $-1$ to 1. With standardization, the regression slopes become the partial correlation coefficient, as correlation is effectively the slope we get when both the response and predictors have been reduced to dimensionless “z-scores.” The PACF plot is used in conjunction with the ACF plot to identify the order of an autoregressive (AR) model. While the ACF helps in understanding the overall autocorrelation structure, the PACF helps pinpoint the specific lags that should be included in the AR component of an ARMA model, ensuring a more accurate estimation.

In summary, the ACF and PACF are powerful tools that enable a deeper understanding of time-series data, guiding the development of robust and effective forecasting models. Their combined use allows for the precise identification of temporal structures, leading to improved predictions and better decision-making in fields where time-series data is prevalent. Figure 2 shows an example of ACF and PACF plots for the same underlying data. The shaded area in the plot represents an approximate confidence interval around zero correlation. In other words, it is a visual guide for checking which autocorrelation (or partial autocorrelation) lags are statistically significant from zero. We can make these plots using the following code:

Figure 2 Left: ACF plot; Right: PACF plot.

Figure 2Long description

In the left graph, the x-axis ranges from 0 to 25 with increments of 5, and the y-axis ranges from minus 1.00 to 1.00. A horizontal line is drawn at 0.00 on the y-axis, with several vertical lines extending upward and downward from it. In the right graph, the x-axis ranges from 0 to 25 with increments of 5, and the y-axis ranges from minus 1.00 to 1.00. A horizontal line is drawn at 0.00 on the y-axis, with multiple vertical lines extending both above and below the horizontal line.

Code 1.2

Code 1.2

1 
2 
3
4 
5 
6 
7 

Next, we discuss another concept that is commonly used in time-series known as stationarity. Stationarity refers to the statistical property of a time-series where its key characteristics, such as mean, variance, and autocovariance structure, remain constant over time. This consistency makes stationary time-series easier to analyze and model, as their behavior is predictable and stable. In financial markets, where data often exhibit complex patterns, achieving stationarity is helpful for accurate modeling, forecasting, and risk management.

There are two forms of stationarity: strict stationarity and weak stationarity. We say that a time-series process is strictly stationary if the joint distribution $f(X_{t_1},\cdots ,X_{t_k})$ is identical to the joint distribution $f(X_{t_1+\tau },\cdots ,X_{t_k+\tau })$ for all collections $t_1,\cdots ,t_k$ and separate values $\tau$. However, this assumption is very restrictive and very few real-world examples meet this requirement. Differently, we say that a time-series process is weakly stationary if:

• $E(X_t)=\mu <\mathrm{\infty }$ where the mean of a time-series is constant and finite,

• $Var(X_t)={\sigma }^2<\mathrm{\infty }$ where the variance of a time-series is constant and finite,

• the autocovariance and autocorrelation functions only depend on the lag:

  $\begin{array}{rl}{\gamma }_{t,t+\tau }& =Cov(X_t,X_{t+\tau })={\gamma }_{\tau },\\ {\rho }_{t,t+\tau }& =Corr(X_t,X_{t+\tau })={\rho }_{\tau }.\end{array}$(9)

A considerable number of statistical and econometric models operate under the assumption that the underlying time-series remains stationary. These models depend on the stability of statistical characteristics to generate precise forecasts. In finance, stationarity ensures that historical risk measures, such as volatility and correlation, remain relevant for future periods. In practice, financial time-series frequently display non-stationary characteristics as a result of trends, seasonal patterns, and structural shifts. To achieve stationarity, analysts use various techniques. For example, they might use differencing to subtract previous observation from the current observation with the aim of removing trends and achieving a stationary series. Additionally, they might remove a deterministic trend component from a series (detrending) or apply transformations like logarithms to stabilize the variance.

2.6 Time-Series Models

We have introduced ACF and PACF which can be used to determine the order of AR and ARMA models. But what exactly are these models? In simple words, these are classical time-series models that form a crucial aspect of analyzing sequential data, particularly in fields such as finance, economics, and environmental science. By identifying patterns and correlations in historical price data, we can employ time-series models to develop quantitative trading strategies by exploiting patterns for profit. AR models, for instance, can help in detecting momentum or mean-reversion patterns, which are commonly used in algorithmic trading.

Beyond financial markets, time-series models are vital for economic policy and planning. Central banks and governmental bodies employ these models to predict key economic metrics, including GDP expansion, inflation levels, and unemployment figures. Accurate forecasts help make informed policy decisions that can stabilize the economy and promote growth. Within the spectrum of time-series models, the Autoregressive model (AR), moving average model (MA), and Autoregressive Moving Average model (ARMA) models are among the most fundamental due to their simplicity and effectiveness. These models are capable of capturing the underlying patterns and dynamics of time-series data, making them powerful tools for analysts and researchers aiming to model and forecast temporal data accurately.

The AR is among the most basic and extensively employed time-series models. It describes the current value of a series as a linear aggregate of its past values combined with an independent random error component. If an AR model takes $p$ previous observations, we denote the model as AR($p$) and its functional form is given as follows:

$X_t={\varphi }_1{X}_{t-1}+\cdots +{\varphi }_p{X}_{t-p}+{ϵ}_t,$(10)

where $X_t$ is the value at current time stamp $t$, ${\varphi }_1,\cdots ,{\varphi }_p$ are the model parameters that represent the effects of past values on the output value, and ${ϵ}_t$ is an error term. For a given order $p$, we can fit the model and find the optimal coefficients ${\varphi }_1$ in the same way as we would fit a linear regression, with the lagged values $(X_{t-1},\dots ,X_{t-p})$ being the predictors and $X_t$ the target. In order to decide which order $p$ to use, we can look at the PACF plot. For example, considering the time-series presented in Figure 2, the PACF suggests that an AR(3) model would probably capture most of the dependence, while it might be useful to consider an AR(9) model to capture further dependence.

The AR model captures how the current observations depend on their past values, making it suitable for modeling time-series with strong autocorrelations. This model is particularly effective when the underlying process is driven by its own past values, which can be the case for stock prices or interest rates. For example, an AR($1$) model, where the output depends only on the immediate past observation, is defined as:

$X_t={\varphi }_1{X}_{t-1}+{ϵ}_t,$(11)

where the model suggests that the current value of the time-series is influenced directly by the observation at time $t-1$.

Differently than the AR model, the MA model represents the current value of a time-series as a linear combination of its previous error terms. The MA model of order $q$, symbolized as MA($q$), is defined as:

$X_t=\mu +{ϵ}_t+{\theta }_1{ϵ}_{t-1}+{\theta }_2{ϵ}_{t-2}+\cdots +{\theta }_q{ϵ}_{t-q},$(12)

where $\mu$ is the mean of the series, ${ϵ}_t,\cdots ,{ϵ}_q$ are error terms, ${\theta }_1,\cdots ,{\theta }_q$ are the model parameters that represent the influence of past errors on the current value. The MA model captures the influence of past shocks or disturbances on the current observation, making it useful for modeling time-series with short-term dependencies. This model is effective when the series is subject to random shocks that have a lasting but diminishing impact over time. For instance, an MA($1$) model defines that the observation at time $t$ is only influenced by the immediate past error:

$X_t=\mu +{ϵ}_t+{\theta }_1{ϵ}_{t-1},$(13)

although, in practice, we can include several terms to model output. To decide the order of a MA model, we check the ACF plot and obtain the estimated point at which the correlation diminishes.

The ARMA model integrates features from both AR and MA models, offering a more adaptable and thorough methodology for time-series analysis. The ARMA model with order $(p,q)$, represented as ARMA($p,q$), is defined as:

$X_t=\sum _{j=1}^p{\varphi }_j{X}_{t-j}+{ϵ}_t+\sum _{j=1}^q{\theta }_j{ϵ}_{t-j},$(14)

where an ARMA model proficiently models long-term dependencies via its AR components and addresses short-term disturbances through its MA components.

In financial time-series analysis, precise forecasting is important for informed investment choices and the formulation of effective trading strategies. AR, MA, and ARMA models provide systematic ways to predict future price movements based on historical data. An AR model can forecast future stock prices by considering the past price movements, while an MA model can evaluate the impact of past market shocks on future prices. ARMA models are often used to estimate future volatility, an essential component of pricing derivatives and constructing risk-hedging strategies.

Given that the aforementioned models are linear, they should always serve as a benchmark before testing any of the more complex, nonlinear deep learning models that are described in later sections. Linear models also have the added benefit of being easy to interpret. This helps form a better intuition for any investment ideas, before moving to more powerful but less interpretable deep learning models.

2.7 Extras

Alpha and Beta

In quantitative finance, the notions of alpha and beta are very important to understanding and evaluating the performance of investment strategies. These metrics are derived from the Capital Asset Pricing Model (CAPM) and are used to measure the returns and risk associated with individual assets or portfolios relative to a benchmark, typically a market index. In quantitative trading, where strategies are often driven by mathematical models and algorithms, alpha and beta provide essential insights into the effectiveness and characteristics of trading approaches.

Alpha assesses an investment’s performance relative to a benchmark index. More specifically, it represents the surplus return that an investment or portfolio achieves beyond the expected return predicted by the CAPM. In other words, alpha signifies the additional value that a trader or investment strategy contributes over what is anticipated based on the asset’s systematic risk. Conversely, beta measures an investment’s responsiveness to market fluctuations. It quantifies the relationship between the investment’s returns and those of the overall market or benchmark, indicating the extent to which the investment’s returns are expected to vary in reaction to changes in the market index. Mathematically, we define alpha ($\alpha$) and beta ($\beta$) as:

$\alpha =R_i-[R_f+\beta (R_m-R_f)],$(15)

where $R_i$ is the return of the investment, $R_f$ is the risk-free rate and $R_m$ is the return of the market. A positive alpha signifies that the investment has surpassed the benchmark, whereas a negative alpha indicates underperformance. In quantitative trading, generating alpha is the primary goal as it reflects the ability of a trading strategy to consistently beat its benchmark through superior stock selection, timing, or other factors. Beta values have different meanings. A beta exceeding 1 signifies that the investment is more volatile than the market, indicating it tends to amplify market movements in response to changes. Conversely, a beta below 1 indicates that the asset’s returns are less sensitive to market movements than the market index itself. If an investment has a negative beta, it means that the investment moves inversely to the benchmark. We can also think of beta as the covariance between strategy and market returns scaled by the market’s variance.

A strategy that consistently generates positive alpha is considered successful, as it indicates the ability to surpass the market performance on a risk-adjusted basis. On the other hand, beta helps traders understand the risk profile of their strategies and manage risk exposure to market volatility. For instance, a trader seeking to minimize risk might construct a low-beta portfolio, while one aiming for higher returns might opt for higher-beta assets. By utilizing alpha and beta metrics, quantitative traders can make well-informed decisions and enhance their trading performance.

Volatility Clustering

Volatility clustering is an extensively observed phenomenon in financial markets, characterized by sequences of high volatility periods that are succeeded by similarly high volatility periods, and periods of low volatility that are followed by similarly low volatility periods. This characteristic implies that volatility is not constant over time but instead exhibits temporal dependencies, forming clusters. This is one of the reasons why financial returns deviate from the normal distribution. This observation is known as heteroskedasticity and describes the irregular pattern of the variation of a process.

Figure 3 shows the returns of the S&P 500, and we can clearly see that large returns tend to cluster. This means that large fluctuations in prices tend to occur together, persistently amplifying the amplitudes of price changes. Such behavior contradicts the assumption of constant variance in traditional models like the classical linear regression model and calls for models that can accommodate changing variances, in order to make reliable predictions.

Figure 3 Returns of the S&P 500 over 60 years.

There are two popular models used to capture and analyze volatility clustering: the Autoregressive Conditional Heteroskedasticity (ARCH) model and the Generalized Autoregressive Conditional Heteroskedasticity (GARCH) model. These models help in capturing the changing variance over time and provide a better fit for the distribution of returns. Instead of predicting returns $R_t$, we now model the variance of returns. An ARCH($p$) process of order $p$ is defined as:

$Var(R_t|R_{t-1},\cdots ,R_{t-p})={\sigma }_t^2={\alpha }_0+{\alpha }_1{R}_{t-1}^2+\cdots +{\alpha }_p{R}_{t-p}^2,$(16)

where the variance of the process at time $t$ is determined by observations from the earlier time step. Accordingly, the ARCH model allows for fluctuations in conditional variance over time, effectively capturing volatility clustering.

The GARCH model extends the ARCH model by including past conditional variances into the model, providing a more flexible and parsimonious model for capturing volatility dynamics. We denote a GARCH($p,q$) as:

${\sigma }_t^2={\alpha }_0+\sum _{j=1}^p{\alpha }_j{R}_{t-j}^2+\sum _{j=1}^q{\beta }_j{\sigma }_{t-j}^2,$(17)

where the GARCH model presents a dual dependence that is better at modeling both short-term shocks and sustained persistence in volatility over time.

In practical terms, volatility clustering means that markets experience periods of turmoil and periods of calm. ARCH and GARCH models offer powerful methods for analyzing this phenomenon, enabling more accurate forecasting, risk management, and pricing of financial instruments. By recognizing the temporal dependencies in volatility, these models enable us to better understand market behavior and enhance decision-making in various financial applications.

3.1 Supervised Learning: Regression and Classification

Supervised learning is at the core of machine learning and it is a process that essentially learns, or in other words, fits a mapping between an input and an output. Formally, for a regression task, it maps an input $\mathbf{\text{x}}\in {\mathbb{R}}^d$ to an output $y\in \mathbb{R}$ through a learned function by training on example input-output pairs. We call this collection of example input-output pairs upon which the model is fitted the training set, and it can be expressed as:

$\{({\mathbf{\text{x}}}_1,y_1),({\mathbf{\text{x}}}_2,y_2),\cdots ,({\mathbf{\text{x}}}_N,y_N)\}.$(18)

A supervised learning algorithm infers a function $f$ that best defines the interplay between inputs and outputs by utilizing training data. The inferred function can then be used to make estimates for new inputs. The function $f$ can be as simple as a linear function or it can also be a highly nonlinear function as obtained through deep learning models. During training, the true output values (labels) are available, and our goal is to reduce the differences between the predicted results and these actual labels. In mathematical terms, this reads:

$\text{min }L(\mathbf{\text{y}},\widehat{\mathbf{\text{y}}})\text{with}{\hat{y}}_i=f({\mathbf{\text{x}}}_i),$(19)

where $L$ is a choice of metric, known as a loss function or an objective function, that measures the difference between real outputs ($\mathbf{\text{y}}$) and predicted values ($\widehat{\mathbf{\text{y}}}$). After learning a functional mapping on the training set, we apply it to unseen test data $\{{\mathbf{\text{x}}}_1^{\prime },{\mathbf{\text{x}}}_2^{\prime },\cdots ,{\mathbf{\text{x}}}_M^{\prime }\}$ and evaluate the performance of our learned function. In general, a supervised learning problem goes through the following steps:

1. 1. Define the prediction problem,

2. 2. Gather a training set that is representative of the application domain,

3. 3. Carry out an exploratory analysis and select input features,

4. 4. Choose the approximate learning algorithm and decide the model’s architecture,

5. 5. Conduct model training on the training set and optimize hyperparameters using a separate validation set,

6. 6. Assess the effectiveness of the trained function using a test dataset.

Depending on outputs $y$, we can divide supervised learning algorithms into two categories: regression and classification. When the output ($y$) takes continuous values, it is a regression problem. For example, stock prices and the weights and heights of a person are all examples of continuous values that would correspond to a regression task. A classification problem deals with discrete outputs, such as whether an image contains a dog or not. The training framework for regression and classification is very similar, with the exception of the design of the objective function. We now discuss each problem type in detail.

Regression

One key aspect of supervised learning is the selection of an appropriate loss function – also referred to as a cost or objective function – that measures the discrepancy between a model’s predictions and the corresponding true target values. The loss function guides the learning process by providing a measure that the model aims to minimize during training. As previously noted, regression problems focus on the prediction of a continuous variable. For regression problems, one of the most commonly used objective functions is the mean-squared error (MSE):

$L(\mathbf{\text{y}},\widehat{\mathbf{\text{y}}})=\frac{1}{N}\sum _{i=1}^N(y_i-{\hat{y}}_i{)}^2,$(20)

where the loss is merely the sum of residuals, ${ϵ}_i=y_i-{\hat{y}}_i$, squared which we aim to minimize to obtain a good fit to the data.Footnote ⁵ The MSE loss is symmetric and places greater emphasis on larger errors in the dataset.

There are numerous options for objective functions. For example, the mean-squared logarithmic error can be applied to outputs that exhibit exponential growth, imposing an asymmetric penalty that is less harsh on negative errors than on positive ones. Both the mean absolute error (MAE) and the median absolute error (MedAE) are symmetric and do not assign additional weight to larger errors. Moreover, Huber loss, which merges aspects of the mean squared error and the mean absolute error, is resistant to outliers and can be used to stabilize training when working with noisy data. Table 1 summarizes some common loss functions for regression problems. It is also very straightforward to implement these losses:

Table 1Objective functions for regression problems.

Table 1Long description

The table consists of two columns Metrics and Formula. It reads as follows: Row 1. Root mean squared error (RMSE); root of 1 divided by N summation subscript i and superscript N (y subscript i minus y cap subscript i) the whole square. Row 2. Mean squared log error (MELE); 1 divided by N summation subscript i and superscript N (ln (1 plus y subscript i) minus ln (1 plus y cap subscript i)) the whole square. Row 3. Mean absolute error (MAE); 1 divided by N summation subscript i and superscript N modulus of y subscript i minus y cap subscript i. Row 4. Median absolute error (MedAE); median (modulus of y subscript i minus y cap subscript i to modulus of y subscript N minus y cap subscript N). Row 5. Huber loss (HL); 1 divided by N summation subscript i and superscript N 1 by 2 (y subscript i minus y cap subscript i) the whole square, if modulus of y subscript i minus y cap subscript i less than or equals delta and 1 divided by N summation subscript i and superscript N delta modulus y subscript i minus y cap subscript i minus 1 by 2 delts square. otherwise.

Code 1.3

Code 1.3

1 
2 
3 
4 
5 
6 
7
8 
9 
10 
11 
12
13 
14 
15 
16 
17 
18 

The best choice of objective function depends on the specific task. Sometimes, we can create customized loss functions to ensure that performance metrics best reflect the consequences of incorrect predictions. For example, in applications like medical diagnosis or fraud detection, false negatives may be more costly than false positives. In such cases, loss functions can be tailored to penalize certain types of errors more severely, aligning the model’s training with the specific needs of the problem.

Classification

Unlike regression, classification aims to place input data into predefined categories or classes. It does so by analyzing a labeled dataset where each example is matched with a class label. Once trained, a classification model can predict labels for unseen data based on the learned patterns and relationships. Classification techniques are commonly applied in fields such as finance, healthcare, and marketing. Classification problems can be broadly categorized into binary classification and multi-class classification. Binary classification involves two distinct classes. Common examples of binary classification include determining whether an email is spam, predicting if a credit card transaction is fraudulent, or diagnosing a patient as healthy or ill. Multi-class problems involve more than two classes, such as classifying handwritten digits (0–9), categorizing types of flowers (e.g., the Iris dataset), or classifying news articles into different topics.

Since classification problems have discrete outputs, we first have to produce scores, or logits, to indicate the likelihoods that an observation belongs to certain classes. These scores can then be normalized across all possible class labels to obtain corresponding probabilities ${\hat{p}}_i$. After that, we use these scores or probabilities ${\hat{p}}_i$ to make actual predictions ${\hat{y}}_i$ either by taking the class with the highest score or by using threshold values. In the simplest case of a binary classification problem, we first define the logistic function

$\sigma (z)=\frac{1}{1+e^{-z}},$(21)

which squashes any real-valued input into the open interval (0,1). We can then model the probability of the prediction being positive as

$p(y_i=1|{\mathbf{\text{x}}}_i)=\sigma ({\mathbf{\text{w}}}^T{\mathbf{\text{x}}}_i).$(22)

The same mapping can be represented by

$\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}({\hat{p}}_i)=log\frac{{\hat{p}}_i}{1-{\hat{p}}_i}={\mathbf{\text{w}}}^T{\mathbf{\text{x}}}_i,$(23)

where the logit of the probabilities is a linear function of input features (${\mathbf{\text{x}}}_i$). The objective functions for classification problems are different from regression as there are instead finite distinct outcomes. In most cases, we choose cross-entropy loss as the objective function for classification. In the binary case, the cross-entropy is calculated as:

$-(ylog(p)+(1-y)log(1-p)),$(24)

where $p$ is the probability of predicting $y=1$. The loss function is then computed by summing the cross-entropy of each data point:

$L(\mathbf{\text{y}},\widehat{\mathbf{\text{p}}})=-\sum _{i=1}^N(y_{i}log({\hat{p}}_i)+(1-y_i)log(1-{\hat{p}}_i)).$(25)

The functional form of the cross-entropy loss might be less intuitive than the MSE, but it can still be understood within the context of maximum likelihood estimation.Footnote ⁶ If we deal with a multi-class classification problem ($M>2$), a separate loss is needed for each class label and a summation is taken at the end:

$-\sum _{c=1}^M{y}_{oc}log(p_{oc}),$(26)

where $y_{o,c}$ is a binary indicator (0 or 1) that is activated when the model assigns the right label $c$ for observation $o$, and $p$ is the output from the algorithm which indicates the predictive probability of observation $o$ for class $c$. The loss of the data is then obtained by summing the multi-class cross-entropy of each point.

Once the predicted probabilities are transformed into predictions of one class or another, we can evaluate model performance through several metrics. To illustrate those metrics we focus on binary classification problems for simplicity. A frequently employed measure is the misclassification rate which can be defined as the fraction of misclassified labels:

$\text{Misclassification rate}=\frac{1}{N}\sum _{i=1}^N{I}_{y_i\ne \widehat{y_i}}.$(27)

The confusion matrix is another important tool that can be used to visualize various metrics. Table 2 illustrates a confusion matrix that enumerates the quantities of correct and incorrect predictions for every class. For example, the False Positives in the top right corner represent errors where an actual label is negative but a prediction is positive. In the context of a stock price reversion example, this would be a case when a stock price does not revert but we predicted that it would revert. Such an error is much more costly to us than a False Negative, where a stock does actually revert but we predicted it would not.

Table 2Confusion matrix.

Table 2Long description

The table consists of three columns blank, Actual: Positive, and Actual: Negative. It reads as follows. Row 1. Prediction: Positive; True Positive (TP); False Positive (FP). Row 2. Prediction: Negative; False Negative (FN); True Negative (TN).

Following the notation in the confusion matrix, we can thus introduce other popular evaluation metrics, which are shown in Table 3. For instance, accuracy is computed by summing the diagonal entries in the confusion matrix and then dividing by the total number of predicted samples. Accuracy thus represents the proportion of total predictions that are correct. Precision indicates the fraction of predicted positives that are truly positive, while recall measures the fraction of actual positives correctly identified. Lastly, the F1 score balances precision and recall by using their harmonic mean.

Table 3Evaluation metrics for classification problems.

Table 3Long description

The table consists of two columns Metrics and Formula. It reads as follows. Row 1. Accuracy; True Positive plus True Negative the whole divided by True Positive plus False Positive plus True Negative plus False Negative. Row 2. Precision; True Positive divided by True Positive plus False Positive. Row 3. Recall; True Positive divided by True Positive plus False Negative. Row 4. F1; 2 multiply Precision multiply recall the whole divided by precision plus recall.

It is very important to check all evaluation metrics when analyzing model performance since a single performance metric can indicate misleading results. For example, in an unbalanced data set, where 90% of labels are +1, we can get an accuracy score of 90% by simply predicting everything as +1, even though the model has not learned anything. Another issue arises when we assign different importance to different types of errors. For example, a mean reversion strategy usually makes frequent small gains but can make infrequent large losses when a stock does not revert. Such a strategy might demonstrate a high accuracy for predicting stock reversion but still lead to significant losses. To implement these metrics, we can use the following code:

Code 1.4

Code 1.4

1 
2
3 
4 
5 
6 
7
8 
9 
10 
11 
12 
13 

Instead of using numerical values to assess model performance, we can also use graphical tools. The receiver operating characteristics (ROC) curve enables us to compare and choose models that are conditioned on their respective predictive performance. For this purpose, we need to compare predicted probabilities with selected thresholds to decide final outcomes. Accordingly, different thresholds yield different results. The ROC curve derives pairs of true positive rates (TPR) and false positive rates (FPR) by examining every possible threshold for classification, and then displays these pairs on a unit square plot. We define TRP and FPR as the following:

$\begin{array}{rl}TPR& =\frac{TP}{TP+FN},\\ FPR& =\frac{FP}{FP+TN}.\end{array}$(28)

Random predictions, on average, yield a diagonal line on the ROC curve which has equal TPR and FPR rates. This diagonal line is the benchmark case, so if the curve falls on the left side of the diagonal line, the learned model is better than random guessing. The further from the margin, the better the classifier (shown in Figure 4). We refer to the area under the ROC curve as AUC and it is a summary measure that tells how good a classifier is. A higher AUC score indicates a better algorithm.

Figure 4 An example of different ROC curves.

Figure 4Long description

The x-axis represents False positive rate, ranging from 0.0 to 1.0 with increments of 0.2 and the y-axis represents True positive rate, ranging from 0.0 to 1.0 with increments of 0.2. The graph displays two curves and one dashed line. The data are as follows. Classifier 1 (area equals 0.73): (0.0, 0.0), (0.4, 0.7), (0.6, 0.8), (1.0, 1.0). Classifier 2 (area equals 0.78): (0.0, 0.1), (0.4, 0.75), (0.6, 0.9), (1.0, 1.0). Random classifier (dashed line): (0.0, 0.0), (0.4, 0.3), (0.6, 0.5), (1.0, 1.0). All the values are approximate.

3.2 Fully Connected Networks

After reviewing the basics of supervised learning, we now look at canonical examples of neural network architectures. Fully connected networks (FCNs), also known as multilayer perceptrons, are one of the earliest and most basic neural networks. FCNs are indispensable in the field of deep learning, as they provide the foundational architecture upon which other more sophisticated neural network models are built. FCNs have powerful abilities to approximate complex functions and patterns within data, making them highly versatile and applicable across various domains.

To gain a thorough understanding of FCNs, let us start with a simple linear model, as a linear model can be viewed as a single neuron with a fully connected input layer. Suppose we have an input of vector form $\mathbf{\text{x}}\in {\mathbb{R}}^{N_x}$ and a scalar output $y\in \mathbb{R}$. A linear model posits that the prediction for an input vector $\mathbf{\text{x}}$ can be determined by:

$y={\mathbf{\text{w}}}^T\mathbf{\text{x}}+b.$(29)

This is precisely how a single neuron (in many neural network frameworks) computes its output – via a linear combination of inputs. In neural networks, this single linear neuron can be extended by stacking many such layers (and adding nonlinearities) to get more expressive models. However, at its core, a single neuron’s linear component is identical to the linear regression formula. For linear regression under a least squares objective, we can write the objective function as:

$L=\frac{1}{N}\sum _{i=1}^N(y_i-{\hat{y}}_i{)}^2,$(30)

where $N$ is the number of sample points. We can therefore optimize the model parameters by setting the partial derivatives of the $L$ with respect to $\mathbf{\text{w}}$ and $b$ to 0:

$\frac{\mathrm{\partial }L}{\mathrm{\partial }\mathbf{\text{w}}}=0\text{ and }\frac{\mathrm{\partial }L}{\mathrm{\partial }b}=0,$(31)

and the aforementioned can be solved analytically. In fact, by setting $b=0$ for simplicity, and writing $X=({\mathbf{\text{x}}}_1,\dots ,{\mathbf{\text{x}}}_N{)}^T$, the solution can easily be obtained as:

$\mathbf{\text{w}}=(X^{T}X{)}^{-1}{X}^T\mathbf{\text{y}}.$(32)

Moreover, one can directly recover the general case with $b\ne 0$ by noting that we can always interpret the bias $b$ as a weight $w_0$ of a constant predictor ${\mathbf{\text{x}}}_0=\mathbf{\text{1}}$. While the analytical solution provides a direct method to find the optimal parameters, it may not be practical for large-scale problems as numerical matrix inversion can become inefficient. As an alternative, gradient descent offers an iterative approach to approximate the optimal parameters by minimizing the MSE:

$\begin{array}{rl}\mathbf{\text{w}}& =\mathbf{\text{w}}-\alpha \frac{\mathrm{\partial }L}{\mathrm{\partial }\mathbf{\text{w}}},\\ b& =b-\alpha \frac{\mathrm{\partial }L}{\mathrm{\partial }b},\end{array}$(33)

where $\alpha$ is the learning rate. This is a rather simple application of gradient descent, especially given that the problem is convex. However, even for a simple linear regression it can be beneficial to use gradient descent. For example, using online gradient descent is a viable strategy for solving cases where the data is so large that it does not fit in memory. Moreover, it is very easy to substitute different loss functions, such as MAE or Huber loss in Table 1.

Note that the notion of learning parameters from data via gradient descent is also the core concept behind more complex neural network training. Understanding how gradients are computed and used to update parameters in linear regression aids in comprehending backpropagation in neural networks. Once again, a single linear neuron is essentially a linear regression with optional activation (defined in the next page). The extension to multiple neurons and stacking them is the essence of deep neural networks.

One of the significant advantages of FCNs is their capacity for universal approximation. According to the Universal Approximation Theorem (Hornik, Stinchcombe, & White, Reference Hornik, Stinchcombe and White1989), a single-hidden-layer feed-forward network with an adequately large neuron count can approximate any continuous function. This property makes FCNs incredibly powerful for tasks involving function approximation. Further, many state-of-the-art models, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), incorporate fully connected layers as integral components. In CNNs, for instance, FCNs are used in the final stages to consolidate the features from the last hidden layers to make predictions.

In general, FCNs receive an input of vector form $\mathbf{\text{x}}\in {\mathbb{R}}^{N_x}$ and map it to an output (here a scalar) $y\in \mathbb{R}$ through a function $y=f(\mathbf{\text{x}}|\mathit{\theta })$. The vector $\mathit{\theta }$ comprises all model parameters, which we iteratively update to achieve the optimal function approximation. Additionally, an FCN can form a chain structure by stacking multiple layers sequentially. Each layer is a function of the previous layer. The first layer can be defined as:

${\mathbf{\text{h}}}^{(1)}=g^{(1)}({\mathbf{\text{W}}}^{(1)}\mathbf{\text{x}}+{\mathbf{\text{b}}}^{(1)}),$(34)

where ${\mathbf{\text{h}}}^{(1)}\in {\mathbb{R}}^{N_1}$ designates the first hidden layer, containing $N_1$ neurons. Additionally, the quantities ${\mathbf{\text{W}}}^{(1)}\in {\mathbb{R}}^{N_1\times N_x}$ and ${\mathbf{\text{b}}}^{(1)}\in {\mathbb{R}}^{N_1}$ denote the associated weight matrix and bias vector, respectively. The function $g^{(1)}(\cdot )$ is called the activation function. We then define the following hidden layers as:

${\mathbf{\text{h}}}^{(l)}=g^{(l)}({\mathbf{\text{W}}}^{(l)}{\mathbf{\text{h}}}^{(l-1)}+{\mathbf{\text{b}}}^{(l)}),$(35)

where the $l$-th hidden layer (${\mathbf{\text{h}}}^{(l)}\in {\mathbb{R}}^{N_l}$) has weights ${\mathbf{\text{W}}}^{(l)}\in {\mathbb{R}}^{N_l\times N_{l-1}}$ and biases ${\mathbf{\text{b}}}^{(l)}\in {\mathbb{R}}^{N_l}$. To better illustrate this, we present an example of an MLP in Figure 5. At its core, each hidden layer computes a linear transformation of the previous layer’s output, followed by a nonlinear activation. The ultimate output is determined by the target’s nature and is once again derived from the preceding hidden layer. The discrepancy between the model’s predictions and the true targets is quantified using a specified loss or objective function. Gradient descent is then employed to adjust the model parameters in an effort to minimize this loss. We can easily build a fully connected network with Pytorch using the following code snippet:

Figure 5 An FCN with two hidden layers in which each hidden layer has five neurons.

Code 1.5

Code 1.5

1 
2
3 
4 
5 
6 
7 
8 
9 
10 
11 
12 
13 
14 
15 
16 

The activation function requires special attention, as its argument is just a linear combination of the model inputs. Therefore, if the activation function is also linear then the overall function represented by the neural network would also be linear. Thus, making the activation function nonlinear is what allows us to represent complex nonlinear functions with neural networks.Footnote ⁷

A variety of activation functions exist and general choices include hyperbolic tangent function, sigmoid function, Rectified Linear Units (ReLU) (Nair & Hinton, Reference Nair and Hinton2010), and Leaky Rectified Linear Units (Leaky-ReLU) (Maas et al., Reference Maas, Hannun and Ng2013). Figure 6 plots some of these activation functions. The ReLU function is prevalent in modern applications, and empirical research advises initiating experimentation with ReLU while simultaneously evaluating other activation functions (Mhaskar & Micchelli, Reference Mhaskar and Micchelli1993). Leaky-ReLUs can also be used to avoid some of the gradient issues caused by the flat part of the ReLU. In broad terms, the selection of activation function to use is dictated by the application context and must be substantiated through validation studies. The same rationale for choosing activation functions likewise applies to other network hyperparameters.

Figure 6 Plots of various activation functions.

3.3 Convolutional Neural Networks

Convolutional neural networks (CNNs) constitute a class of deep learning architectures meticulously engineered to process images and other structured grid-like data. These networks have fundamentally changed the landscape of computer vision, allowing machines to carry out tasks such as image recognition, object detection, and image segmentation with performance levels that rival human capabilities. Drawing inspiration from the visual cortex of animals, the architecture of CNNs enables them to automatically and adaptively learn spatial hierarchies of features from input data, thereby enhancing their effectiveness in various pattern recognition tasks.

CNNs are arguably the most important network structures as they inspired much of the development of modern deep learning algorithms over the past decade through initial breakthroughs in performance on image recognition problems. Marking a pivotal breakthrough in computer vision, Krizhevsky, Sutskever, and Hinton (Reference Krizhevsky, Sutskever and Hinton2017) introduced the first convolutional neural network successfully applied to large-scale image tasks. Often in image problems, features learned by neural networks have intuitive interpretations such as edges or surfaces. Whereas an FCN would need to relearn a feature for every part of an image, the CNNs architecture enables the model to learn the same feature for different parts of an image by sliding a convolutional filter across it.

Subsequently, CNNs were studied and applied to many domain areas. We now demonstrate how the same concept can be used for time-series problems. Time-series data, characterized by its sequential and temporal nature, can benefit from the unique ability of CNNs to detect patterns and trends over different scales. By adapting the convolutional operations used in image processing to time-series data, CNNs can effectively capture local dependencies and extract representative features. These qualities make them strong tools for time-series forecasting, anomaly detection, and classification.

Unlike MLPs that receive inputs in vector format, CNNs are adept at processing grid-structured input data through the use of two specialized layer types: convolutional layers and pooling layers. Convolutional layers constitute the primary components of a CNNs, with each convolutional layer containing multiple convolutional filters designed to extract local spatial relationships from the input data. Convolutional filters, also known as kernels or feature detectors, are designed to traverse and transform input data by detecting specific features or patterns. In essence, a convolutional filter is a diminutive weight matrix that slides across the input data, performing a dot product with each localized region of the input. This procedure is referred to as the convolution operation. We denote a standard convolutional filter as $\mathbf{\text{K}}$ and it processes the input data $\mathbf{\text{X}}\in {\mathcal{R}}^{N_T\times N_x}$ by utilizing a convolution operation:

$\mathbf{\text{S}}(i,j)=(\mathbf{\text{X}}*\mathbf{\text{K}})(i,j)=\sum _{m=0}^{M-1}\sum _{n=0}^{N-1}\mathbf{\text{X}}(i+m,j+n)\mathbf{\text{K}}(m,n),$(36)

where $\mathbf{\text{S}}$ signifies the resultant matrix (feature map) and $(i,j)$ correspond to the indices of its rows and columns. We denote the convolution process as $*$.

A single convolutional layer is capable of containing multiple filters, each of which convolves the input data using a distinct set of parameters. The matrices produced by these filters are often termed feature maps. Similar to MLPs, these feature maps can be transmitted to subsequent convolutional layers and subjected to activation functions to incorporate nonlinearities into the model. In time-series modeling, the primary strategy involves applying convolutional filters along the temporal axis, thereby enabling the network to discern and learn temporal dependencies and patterns.

Another crucial component of a CNNs is the pooling layer, which also features a grid-like structure. This layer condenses the information from specific areas of the feature maps by applying statistical operations to nearby outputs. For example, the widely used max-pooling layer (Y.- T. Zhou & Chellappa, Reference Zhou and Chellappa1988) selects the highest value within a designated region of the feature maps, whereas average pooling computes the mean value of that region. The study by Boureau, Ponce, and LeCun (Reference Boureau, Ponce and LeCun2010) explores the application of various pooling methods in different contexts. However, in most scenarios, selecting the appropriate pooling technique necessitates domain expertise and empirical experimentation.

Pooling layers are utilized across various applications to make the resulting feature maps relatively invariant to small changes in the input data. This type of invariance is beneficial when the focus is on detecting the existence of particular features rather than their exact positions (Goodfellow, Bengio, & Courville, Reference Goodfellow, Bengio and Courville2016). For instance, in certain image classification problems, it is only necessary to recognize that an image contains objects with specific characteristics without needing to pinpoint their exact locations. Conversely, in time-series analysis, the precise timing or placement of features is often essential, and therefore the use of pooling layers must be approached with caution.

In addition to convolutional and pooling layers, a CNNs has additional possible operations: padding and stride. Padding is employed to preserve the dimensions of the feature maps, as convolution operations would otherwise “shrink” the dimension of original inputs (demonstrated in Figure 7). Padding solves this by adding, or “padding,” the original inputs with zeros around the borders (zero-padding) so that the resulting feature maps have the same dimension as before (the top-right figure of Figure 7).

Figure 7 Top: an illustration of padding; Bottom: an illustration of stride.

Separately, stride is a commonly utilized parameter that controls how the convolutional filter moves across inputs and can reduce the dimensionality of input data. More concretely, stride defines the number of steps the filter takes as it slides over the input matrix, impacting the dimension of the output feature map and the amount of computational work required. Recall that a convolutional filter “scans” an input and by default moves by a step size of (1,1). Stride defines a different step size. For example, a stride of (2,2) has the effect of moving the filter two steps and thus decreases the original input by half (as shown in Figure 7).

Padding and stride are central concepts within CNNs that manage the spatial dimensions of output feature maps and determine the network’s effectiveness in capturing and retaining information from the input data. By carefully choosing padding and stride values, one can balance the trade-offs between computational efficiency and the level of detail captured in the features extracted by the network. Finally, we can combine all of these components to construct a convolutional network. Figure 8 shows a typical example of a CNNs and possesses an architecture that is standard and highly popular in image classification problems. Other famous networks for further independent study include “AlexNet” (Krizhevsky, Sutskever, & Hinton, Reference Krizhevsky, Sutskever and Hinton2012) and “VGGNet” (Simonyan & Zisserman, Reference Simonyan and Zisserman2014).

Figure 8 An example CNNs network that first goes through a convolutional layer and then a pooling layer with a fully connected layer at the end.

3.4 WaveNet

CNNs are naturally desirable for dealing with stochastic financial time-series as convolutional layers have smoothing properties that facilitate the extraction of valuable information and discard the noise. In addition, a convolutional filter can be configured to have fewer trainable weights than fully connected layers. To some extent, this remedies the problem of overfitting (defined in Section 4). However, a convolutional filter summarizes information for local regions of the input, so the receptive field from convolutional layers is limited, and in order to consider the entire input sequence, we have to use many layers and such operations could be highly inefficient.

We now introduce WaveNet, an architecture that specifically addresses this issue by using dilated convolutions. A WaveNet is a deep generative model developed by DeepMind (Van Den Oord et al., Reference Van Den Oord, Dieleman and Zen2016) which generates raw audio waveforms and represents a substantial milestone in audio synthesis and processing. It is capable of producing highly realistic human speech and other audio signals by directly modeling the raw waveform of the audio, unlike traditional methods that rely on intermediate representations such as spectrograms or parametric models.

WaveNet has proven to be a powerful tool that can be effectively adapted for time-series analysis. Time-series data, characterized by sequential and temporal dependencies, presents a set of challenges that a WaveNet architecture is well-suited to address. By leveraging its strengths to study prolonged dependencies and capture intricate temporal patterns, A WaveNet offers a robust framework for modeling time-series processes. The core of a WaveNet is the dilated causal convolutions which enable the network to consider a broad context of past observations. In essence, the dilated convolutions skip some elements in the input, which allows the networks to access a larger range of inputs. Apart from dilated convolutional layers, other important components of a WaveNet include residual and skip connections and gated activation units.

The work of Borovykh, Bohte, and Oosterlee (Reference Borovykh, Bohte and Oosterlee2017) proposes that the dilated convolution for a time-series has a large receptive field. Specifically, a dilated convolution is defined as:

$(w_h^l{*}_d{f}^{l-1})(i)=\sum _{j=-\mathrm{\infty }}^{\mathrm{\infty }}\sum _{m=1}^{M_{l-1}}{w}_h^l(j,m)f^{l-1}(i-d\cdot j,m),$(37)

where $M_l$ denotes the number of channels and $d$ is the dilation factor. A dilated convolutional filter operates with every $d$ th element in the input, therefore, it can access a broad range of inputs compared to standard convolutional filters. The causal nature of the convolutions ensures that the model does not violate the temporal order of the time-series, making it suitable for prediction tasks.

We can stack multiple such layers to extract even longer dependencies. For a network with $L$ dilated convolutional layers, we increase the dilation factor by two at each layer, so that $d\in [2^0,2^1,\cdots ,2^{L-1}]$, and the filter size $w$ is $1\times k:=1\times 2$. As a result, the dilation rate exponentially increases with each layer, allowing the network to efficiently model prolonged dependencies over sequences. An example of a dilated convolutional network that consists of three layers is illustrated in Figure 9. To incorporate nonlinearity into the model, we use activation functions after each layer to transform the resulting representations. A WaveNet that uses ReLU has output from layer $l$ is:

Figure 9 A WaveNet with three layers. The dilation factors for the first, second, and third hidden layers are 1, 2, and 4 respectively.

$f^l=[ReLU(w_1^l{*}_d{f}^{l-1})+b,\cdots ,ReLU(w_{M_l}^l{*}_d{f}^{l-1})+b],$(38)

where ${*}_d$ refers to a convolution performed with a dilation factor of $d$, $b\in \mathbb{R}$ is the bias term, and $f^l$ indicates the output generated from a convolution using the filters $w_l^h$ for each $h=1,\cdots ,M_l$ within layer $l$.

In general, a deep network can suffer from an unstable training process if backpropagation becomes unstable during the process of differentiation across multiple layers. This problematic phenomenon is called the degradation problem and was discussed by He, Zhang, Ren, and Sun (Reference He, Zhang, Ren and Sun2016). The work of He et al. (Reference He, Zhang, Ren and Sun2016) proposes residual connections to solve this limitation by forcing the network to approximate $\mathcal{H}(x)-x$ instead of $\mathcal{H}(x)$ (the outputs from an intermediate layer). They suggest that optimizing the residual mapping is easier and they implement this technique by adding the inputs and outputs from a neural layer together. In a WaveNet (Figure 10), each dilated convolutional layer is followed by a residual connection. Specifically, the outputs from the activation function undergo a $1\times 1$ convolution (a point-wise convolution) before the residual connection is applied. This approach ensures that both the residual path and the output from the dilated convolution have the same number of channels, allowing multiple layers to be stacked effectively. Finally, we can use WaveNet as an autoregressive model for time-series forecasting. In this context, the expectation for predicting every $t\in \{0,\dots ,N\}$ is:

Figure 10 The network structure of a WaveNet. The input is convolved in the first layer and then fed to the following network layer with a residual connection. The Condition refers to any other external information that the network uses. This operation is repeated until the output layer $L(M)$ and the final forecast is made.

Figure 10Long description

Two parallel branches begin from Input and Condition. Each branch has an upward arrow connecting to its own Dilated Convolution layer. These are followed by upward arrows leading to individual ReLU. From both Input and Condition paths, connections lead to a central circle with a plus icon. The output of this summation goes to Output Layer 1. Below this layer, there are labels on both sides reading Parametrized skip connection. Two branches originate from Output Layer I-1 and Output Layer L. The Output Layer I-1 branch flows upward through a Dilated Convolution followed by a ReLU layer. This branch connects to a central summation circle (plus icon), which outputs to Output Layer I. A label reading Residual connection appears below this layer. The second branch from Output Layer L passes through a 1 multiply 1 Convolution and continues upward to a block labeled Y. A dashed arrow also connects Output Layer L back to Output Layer I.

$\mathbb{E}[x_{t+1}|x_t,\cdots ,x_{t-r}]={\beta }_1{x}_{t-r}+\cdots +{\beta }_r{x}_t,$(39)

where ${\beta }_i$ are parameters optimized via gradient descent. To create a one-step-ahead prediction, we compute ${\hat{x}}_{t+1}$ for $t+1\ge r$ by inputting the sequence $(x_{t+1-r},\dots ,x_t)$. These predictions can subsequently be reintroduced into the network to formulate $n$-step-ahead forecasts. For instance, a two-step-ahead out-of-sample prediction ${\hat{x}}_{t+2}$ is produced by using the input $(x_{t+2-r},\dots ,{\hat{x}}_{t+1})$.

3.5 Recurrent Neural Networks

Recurrent neural networks (RNNs) form a class of neural architectures specifically devised to detect patterns within sequential data, including time-series, natural language, and speech signals. In contrast to conventional feed-forward neural networks which presume independence among inputs, RNNs possess an internal memory structure that captures details of preceding inputs. This design feature enables RNNs to maintain context and appreciate temporal dependencies, rendering them highly suitable for scenarios where the sequential order and context of data are integral.

RNNs have a rich history that dates back to the early days of artificial intelligence and deep learning research. The foundational idea behind RNNs was to create a network that could process sequences of data and retain a memory mechanism to model time-based dependencies, which traditional feed-forward networks could not handle effectively. Given this idea, time-series data that have sequential structures are well suited to the framework of RNNs.

When dealing with time-series data, we can take previous observations to form inputs. For a multivariate time-series, a single input ${\mathbf{\text{x}}}_{1:T}\in {\mathcal{R}}^{N_T\times N_x}$ has two dimensions where ${\mathbf{\text{x}}}_t$ represents the features corresponding to time $t$, and $T$ is the length of the input’s look-back window. To employ an MLP for modeling these processes, the inputs need to be flattened prior to feeding them into the hidden layers. However, this operation could break the existing time dependencies in inputs. To address this potential problem, RNNs have a structure that maintains an internal representation that preserves important temporal relationships

The RNNs architecture leverages hidden states that function as a memory mechanism, and recursively update with each new observation at every time step. Consequently, this structure naturally carries information forward from earlier inputs to current ones, extracting temporal patterns from the data. We can define a hidden state as:

$\begin{array}{rl}{\mathbf{\text{h}}}_t& =f({\mathbf{\text{h}}}_{t-1},{\mathbf{\text{x}}}_t|\mathbf{\text{W}},\mathbf{\text{b}}),\\ & =g({\mathbf{\text{W}}}_h{\mathbf{\text{h}}}_{t-1}+{\mathbf{\text{W}}}_x{\mathbf{\text{x}}}_t+\mathbf{\text{b}}),\end{array}$(40)

where ${\mathbf{\text{W}}}_h\in {\mathbb{R}}^{N_h\times N_h}$, ${\mathbf{\text{W}}}_x\in {\mathbb{R}}^{N_h\times N_x}$, $\mathbf{\text{b}}\in {\mathbb{R}}^{N_h}$ constitute the linear weights and biases for the hidden state, while $g(\cdot )$ designates the activation function. The quantity $N_h$ corresponds to the number of hidden units, and $N_x$ refers to the number of input features observed at any time $t$. An illustrative example of such an RNNs is depicted in Figure 11.

Figure 11 A recurrent network that processes information from the input and the past hidden state.

Nevertheless, due to the model’s recursive architecture, taking the derivative of the objective function with respect to its parameters involves a sequence of multiplicative terms that could lead to vanishing or exploding gradients for RNNs (Bengio, Simard, & Frasconi, Reference Bengio, Simard and Frasconi1994). This issue complicates the back-propagation of gradients, resulting in an unstable training procedure and limiting RNNs’ effectiveness in modeling long-term dependencies.

A significant breakthrough came in 1997 when Hochreiter and Schmidhuber (Reference Hochreiter and Schmidhuber1997) introduced the Long Short-Term Memory (LSTMs) network. LSTM addressed the vanishing gradient problem by introducing memory cells and gating mechanisms that allowed the model to retain and selectively update information over prolonged sequences. This innovation dramatically improved the ability of RNNs to learn and remember over longer time periods, making LSTMs a crucial development in the field.

Much like an RNN, an LSTM maintains a chain of hidden states that undergo recursive updates. However, the LSTM architecture incorporates an internal memory cell, along with three gates – specifically the input gate, forget gate, and output gate – that oversee how information flows into and out of the cell. These gates facilitate the long-term maintenance and adjustment of the network’s state over lengthy sequences. We define the gates as follows:

$\begin{array}{rl}\text{Input gate:}& {\mathbf{\text{i}}}_t=\sigma ({\mathbf{\text{W}}}_{i,h}{\mathbf{\text{h}}}_{t-1}+{\mathbf{\text{W}}}_{i,x}{\mathbf{\text{x}}}_t+{\mathbf{\text{b}}}_i),\\ & \text{where}{\mathbf{\text{W}}}_{i,h}\in {\mathbb{R}}^{N_h\times N_h},{\mathbf{\text{W}}}_{i,x}\in {\mathbb{R}}^{N_h\times N_x}\text{and}{\mathbf{\text{b}}}_i\in {\mathbb{R}}^{N_h},\\ \text{Output gate:}& {\mathbf{\text{o}}}_t=\sigma ({\mathbf{\text{W}}}_{o,h}{\mathbf{\text{h}}}_{t-1}+{\mathbf{\text{W}}}_{o,x}{\mathbf{\text{x}}}_t+{\mathbf{\text{b}}}_o),\\ & \text{where}{\mathbf{\text{W}}}_{o,h}\in {\mathbb{R}}^{N_h\times N_h},{\mathbf{\text{W}}}_{o,x}\in {\mathbb{R}}^{N_h\times N_x}\text{and}{\mathbf{\text{b}}}_o\in {\mathbb{R}}^{N_h},\\ \text{Forget gate:}& {\mathbf{\text{f}}}_t=\sigma ({\mathbf{\text{W}}}_{f,h}{\mathbf{\text{h}}}_{t-1}+{\mathbf{\text{W}}}_{f,x}{\mathbf{\text{x}}}_t+{\mathbf{\text{b}}}_f),\\ & \text{where}{\mathbf{\text{W}}}_{f,h}\in {\mathbb{R}}^{N_h\times N_h},{\mathbf{\text{W}}}_{f,x}\in {\mathbb{R}}^{N_h\times N_x}\text{and}{\mathbf{\text{b}}}_f\in {\mathbb{R}}^{N_h},\end{array}$(41)

where we define ${\mathbf{\text{h}}}_{t-1}$ as the LSTM’s hidden state at time step $t-1$ and apply the sigmoid activation function $\sigma (\cdot )$. The parameters $\mathbf{\text{W}}$ and $\mathbf{\text{b}}$ represent the model’s weights and biases. The resulting cell state and hidden state at the current time step are then described by:

$\begin{array}{rl}\text{Cell state:}& {\mathbf{\text{c}}}_t={\mathbf{\text{f}}}_t\odot {\mathbf{\text{c}}}_{t-1}+{\mathbf{\text{i}}}_t\odot \text{tanh}({\mathbf{\text{W}}}_{c,h}{\mathbf{\text{h}}}_{t-1}+{\mathbf{\text{W}}}_{c,x}{\mathbf{\text{x}}}_t+{\mathbf{\text{b}}}_c),\\ \text{Hidden state:}:& {\mathbf{\text{h}}}_t={\mathbf{\text{o}}}_t\odot \text{tanh}({\mathbf{\text{c}}}_t),\end{array}$(42)

where $\odot$ is the element-wise product, ${\mathbf{\text{W}}}_{c,x}\in {\mathbb{R}}^{N_h\times N_x}$, ${\mathbf{\text{b}}}_c\in {\mathbb{R}}^{N_h}$, ${\mathbf{\text{W}}}_{c,h}\in {\mathbb{R}}^{N_h\times N_h}$, and $\text{tanh}(\cdot )$ is the hyperbolic tangent activation function. Figure 12 plots an LSTM cell with all gates mechanisms.

Figure 12 An illustration of an LSTM cell.

LSTMs have been applied successfully in numerous fields because of their unique properties for dealing with prolonged sequences. For instance, LSTMs are widely used in language modeling, where they predict the next word in a sequence, as well as in machine translation, where they translate text from one language to another. For financial applications, LSTMs are also well studied and there exists a large amount of literature that applies LSTMs to predict financial time-series. Despite their success, LSTMs still suffer from several issues. Firstly, due to the gating mechanism and cell structure, LSTMs are very complex, which leads to a considerable amount of parameters that must be learned. As a result, the problem of overfitting is severe in certain applications. LSTMs are also computationally intensive, requiring lengthy training schedules.

In an effort to address the complications and drawbacks of LSTMs, Cho et al. (Reference Cho, Van Merriënboer and Gulcehre2014) proposed the Gated Recurrent Units (GRUs) as a more straightforward alternative. GRUs also aim to mitigate the vanishing gradient problem but do so by utilizing a reduced parameter set. This makes them computationally more efficient while often achieving performance on par with LSTMs. Unlike LSTMs, GRUs merge the forget and input gates into an update gate and also combine the cell and hidden states into a single vector. This results in fewer parameters and a leaner design. In a GRU, there are two primary gates: the update gate and the reset gate. We can summarize the operation of a GRU as follows:

$\begin{array}{rl}{\mathbf{\text{z}}}_t& =\sigma ({\mathbf{\text{W}}}_{z,h}{\mathbf{\text{h}}}_{t-1}+{\mathbf{\text{W}}}_{z,x}{\mathbf{\text{x}}}_t+{\mathbf{\text{b}}}_i),\\ {\mathbf{\text{r}}}_t& =\sigma ({\mathbf{\text{W}}}_{r,h}{\mathbf{\text{h}}}_{t-1}+{\mathbf{\text{W}}}_{r,x}{\mathbf{\text{x}}}_t+{\mathbf{\text{b}}}_o),\\ {\tilde{h}}_t& =tanh({\mathbf{\text{W}}}_h({\mathbf{\text{r}}}_t\odot {\mathbf{\text{h}}}_{t-1})+{\mathbf{\text{W}}}_{h,x}{\mathbf{\text{x}}}_t),\\ {\mathbf{\text{h}}}_t& =(1-{\mathbf{\text{z}}}_t)\odot {\mathbf{\text{h}}}_{t-1}+{\mathbf{\text{z}}}_t\odot {\widetilde{\mathbf{\text{h}}}}_t,\end{array}$(43)

where ${\mathbf{\text{z}}}_t$ functions as the update gate, ${\mathbf{\text{r}}}_t$ serves as the reset gate, ${\widetilde{\mathbf{\text{h}}}}_t$ denotes the candidate hidden state, and ${\mathbf{\text{h}}}_t$ corresponds to the new hidden state.

Overall, GRUs feature a more streamlined architecture than LSTMs, making them less complex to implement and quicker to train. With fewer gates and combined states, GRUs have fewer parameters, reducing the risk of overfitting. GRUs offer an efficient alternative that retains the key advantages of LSTMs. Understanding the differences and trade-offs between LSTMs and GRUs allows practitioners to choose the appropriate architecture for their specific needs.

3.6 Seq2seq and Attention

In previous sections, we focus on a single-point estimation. In other words, our model can only make predictions for a fixed horizon that is specified beforehand. If we are interested in predictions at various horizons, several models need to be fitted, with each requiring independent training. However, information flows from past to future for time-series. We could thus expect that features that are meaningful for short-term predictions could be used for long-term predictions. Therefore it would be a waste to treat them independently. Here, we introduce the Sequence-to-Sequence model (Seq2Seq) and the Attention mechanism that enable us to make multi-horizon forecasts. Both models have an encoder-decoder structure and we can simultaneously forecast all horizons of interest.

3.6.1 Sequence to Sequence Learning (Seq2Seq)

In Sutskever, Vinyals, and Le (Reference Sutskever, Vinyals and Le2014), the Seq2Seq model was proposed as a significant advancement in the realm of neural networks, especially for NLP applications. The earliest Seq2Seq framework focused on machine translation, whereby text is transformed from one language into another. Prior approaches, such as statistical machine translation, had difficulty handling intricate language patterns and preserving natural fluency. The Seq2Seq model, with its encoder-decoder architecture, provides a more robust framework for handling such tasks.

A standard Seq2Seq model is comprised of two core components: an encoder that encodes an input sequence into a fixed-dimensional representation, and a decoder that leverages this representation to generate an output sequence. Early Seq2Seq implementations typically employed RNNs for both encoding and decoding. However, these RNN-based models struggled with longer input sequences, largely because of issues like vanishing gradients. This led to the adoption of LSTM networks and GRUs, which provided better handling of long-term dependencies.

Seq2Seq architectures were also soon applied to financial time-series. Notably, Z. Zhang and Zohren (Reference Zhang and Zohren2021) introduced an application of Seq2Seq and Attention models in the context of financial time-series. Consider an input sequence ${\mathbf{\text{x}}}_{1:T}=({\mathbf{\text{x}}}_1,{\mathbf{\text{x}}}_2,\cdots ,{\mathbf{\text{x}}}_T)\in {\mathbb{R}}^{T\times m}$, where each ${\mathbf{\text{x}}}_t\in {\mathbb{R}}^m$ is an $m$-dimensional feature vector at time $t$ and $T$ is the total length of the sequence. The encoder processes these vectors step by step to derive meaningful features, and the resulting context vector captures the relevant information gathered by the encoder. After that, the decoder utilizes the information from the context vector and generates the output ${\mathbf{\text{y}}}_{1:k}=(y_1,y_2,\cdots ,y_k)\in {\mathbb{R}}^{k\times n}$ where $k$ is the furthest prediction point. Specifically, given a single ${\mathbf{\text{x}}}_{1:T}$, we can derive the hidden state (${\mathbf{\text{h}}}_t$) with the previous hidden state and current observations (${\mathbf{\text{x}}}_t$):

$\text{Encoder:}{\mathbf{\text{h}}}_t=f({\mathbf{\text{h}}}_{t-1},{\mathbf{\text{x}}}_t),$(44)

where $f$ can be a simple RNNs model or complex recurrent network. The encoder iterates over the input sequence until it reaches the final time step, and its last hidden state serves as a summary of the entire input. In Seq2Seq models, this final hidden state is often taken as the context vector $c$, functioning as the “bridge” between the encoder and decoder. For the decoder, the hidden state ${\mathbf{\text{h}}}_t^{\prime }$ is defined as:

$\text{Decoder:}{\mathbf{\text{h}}}_t^{\prime }=f({\mathbf{\text{h}}}_{t-1}^{\prime },y_{t-1},\mathbf{\text{c}}),$(45)

and the distribution for output $y_t$ is:

$P(y_t|y_{t-1},y_{t-2},\cdots ,y_1,\mathbf{\text{c}})=g({\mathbf{\text{h}}}_t^{\prime },\mathbf{\text{c}}),$(46)

where $f$ and $g$ can be various functions but $g$ needs to produce valid probabilities, which could be achieved through a softmax activation function (Equation 47). Figure 13 shows an example of a standard Seq2Seq network.

Figure 13 A typical example of a Seq2Seq network.

$\text{softmax}(\mathbf{\text{z}}{)}_i=\frac{exp(z_i)}{\sum _j^{K}exp(z_j)},i=1,\cdots ,K.$(47)

Seq2Seq models have advanced a wide array of tasks in NLP and other fields. In machine translation, Seq2Seq revolutionized the field by providing more accurate and fluent translations compared to traditional methods. These abilities enabled the generation of concise summaries from long documents, aiding in information extraction and content curation. Further, Seq2Seq models powered early chatbots and virtual assistants, allowing for context-aware responses in dialogues.

A primary drawback of traditional Seq2Seq architectures is that they compress the entire input sequence into one fixed-dimensional context vector. For short sequences, this approach works reasonably well. But, for longer sequences, it becomes problematic as the context vector may not encapsulate all the relevant information. This can potentially lead to a loss of important details and degrade the quality of the generated output. Consequently, the decoder can find it challenging to generate precise and coherent outputs, especially for tasks that require retaining detailed information over extended sequences. These shortcomings led to the subsequent development of the attention mechanism.

3.6.2 Attention

The attention mechanism, introduced by Bahdanau, Cho, and Bengio (Reference Bahdanau, Cho and Bengio2014), enables a model to dynamically attend to different parts of an input sequence instead of relying solely upon a single fixed-size context vector. This approach leverages a system of alignment scores, attention weights, and context vectors to provide the model with greater flexibility, thereby improving its ability to handle longer sequences effectively.

In attention-based models, alignment scores are first calculated to assess the relevance of each encoder hidden state to the current decoder state, indicating how each input token influences the token being generated. These scores are then normalized with a softmax function to yield attention weights, which dynamically control how much emphasis each input token receives at each decoding step. Next, a weighted sum of the encoder hidden states is taken according to these attention weights, resulting in a context vector that highlights the most pertinent aspects of the input. This context vector is then used by the decoder to generate the next token in the output sequence.

The attention mechanism also follows an encoder-decoder architecture. We can denote the encoder’s hidden state at time $t$ by ${\mathbf{\text{h}}}_t$:

$\text{Encoder:}{\mathbf{\text{h}}}_t=f({\mathbf{\text{h}}}_{t-1},x_t),$(48)

where $f$ is a non-linear function that is similar to a Seq2Seq model. The difference lies in the decoder structure as we now need to compute attention weights, alignment scores, and context vectors. Specifically, we define the context vector ${\mathbf{\text{c}}}_t$ and attention weights at the time stamp $t$ as:

$\begin{array}{rlrl}& \text{Context vector:}& & {\mathbf{\text{c}}}_t=\sum _{i=1}^T{\alpha }_{t,i}{h}_i,\\ & \text{Attention weight:}& & {\alpha }_{t,i}=\frac{exp(e({\mathbf{\text{h}}}_{t-1}^{\prime },{\mathbf{\text{h}}}_i))}{\sum _{j=1}^{T}exp(e({\mathbf{\text{h}}}_{t-1}^{\prime },{\mathbf{\text{h}}}_j))},\end{array}$(49)

where $e({\mathbf{\text{h}}}_{t-1}^{\prime },{\mathbf{\text{h}}}_i)$ is the attention score that indicates the weights placed by the context vector on each time step of the encoder. The work of Luong, Pham, and Manning (Reference Luong, Pham and Manning2015) introduces three methods to compute the score:

$e({\mathbf{\text{h}}}_{t-1}^{\prime },{\mathbf{\text{h}}}_i)=\left\{\begin{array}{ll}{\mathbf{\text{h}}}_i^{\text{T}}{\mathbf{\text{h}}}_{t-1}^{\prime }& \text{dot},\\ {\mathbf{\text{h}}}_i^{\text{T}}{\mathbf{\text{W}}}_a{\mathbf{\text{h}}}_{t-1}^{\prime }& \text{general},\\ tanh({\mathbf{\text{W}}}_a[{\mathbf{\text{h}}}_i^{\text{T}};{\mathbf{\text{h}}}_{t-1}^{\prime }])& \text{concatenate}.\end{array}\right.$(50)

Finally, similar to the process for a Seq2Seq model, the context vector ${\mathbf{\text{c}}}_t$ is fed to the decoder:

$\begin{array}{rl}\text{Decoder:}{\mathbf{\text{h}}}_t^{\prime }& =f({\mathbf{\text{h}}}_{t-1}^{\prime },y_{t-1},{\mathbf{\text{c}}}_t),\\ P(y_t|y_{t-1},y_{t-2},\cdots ,y_1,{\mathbf{\text{c}}}_t)& =g({\mathbf{\text{h}}}_t^{\prime },{\mathbf{\text{c}}}_t),\end{array}$(51)

where ${\mathbf{\text{h}}}_t^{\prime }$ denotes the hidden state at time $t$ and the activation function is denoted by $g$. An illustrative example of the Attention mechanism is shown in Figure 14.

Figure 14 An example of Attention.

In essence, the attention mechanism was conceived to address the drawbacks of Seq2Seq models – namely, their dependence on a fixed-size context vector and the ensuing information bottleneck. By enabling the model to selectively focus on different regions of the input sequence, attention mechanisms substantially improve the handling of lengthy inputs and the retention of crucial contextual information.

By granting the decoder access to all the encoder hidden states, rather than relying on a single fixed-size context vector, the information bottleneck issue is significantly alleviated. Moreover, attention mechanisms promote better gradient flow during training, helping to mitigate the vanishing gradient problem in RNNs and enhancing the model’s capacity to capture long-range dependencies.

3.7 Transformers

The attention mechanism is very powerful as it enables context vectors to incorporate information across longer sequences. However, such a model possesses a chain structure that is very slow to train. This problem worsens as input lengths increase. To address this issue, the Transformer network was designed by Vaswani et al. (Reference Vaswani, Shazeer and Parmar2017) and represents a major advancement in leveraging attention mechanisms.

Unlike Seq2Seq models which have a recurrent structure that is slow to train, the Transformer architecture introduces a parallelizable attention mechanism that allows it to process entire sequences simultaneously. Transformers substantially reduce training and inference times by leveraging parallel processing, which makes them more efficient when dealing with large datasets. The parallel efficiency of the Transformer model originates from the self-attention mechanism (also referred to as scaled dot-product attention). As the foundational component of the Transformer architecture, self-attention enables the model to evaluate the relationships between all tokens within the input sequence at once, independent of their positional distance. This capability is especially beneficial for capturing long-range dependencies and makes Transformers highly scalable and versatile.

Recent architectures have consistently delivered state-of-the-art results and top-tier performance across a variety of tasks, including language translation, text summarization, and question-answering. Such models can contain billions of parameters and handle extremely large-scale datasets. For example, BERT, introduced by Devlin, Chang, Lee, and Toutanova (Reference Devlin, Chang, Lee and Toutanova2018), was a groundbreaking model in natural language processing that revolutionized the way language models understand text. OpenAI’s GPT series comprises autoregressive language models that have also markedly pushed forward developments in the field of natural language generation.

In this section, we carefully introduce the Transformer designed by Vaswani et al. (Reference Vaswani, Shazeer and Parmar2017). A solid grasp of the foundational Transformer components is crucial, as many cutting-edge models extend from them. A typical Transformer architecture follows an encoder-decoder design. The encoder includes input embedding, positional encoding, and an attention mechanism. Specifically, the attention module consists of a stack of $N$ layers, each containing a multi-head self-attention sub-layer and a position-wise fully connected feed-forward sub-layer. The decoder mirrors this structure with its own stack of layers – often of the same depth as the encoder – but each decoder layer incorporates three main sub-components: masked multi-head self-attention, encoder-decoder attention, and a position-wise feed-forward network. Figure 15 shows various components of transformers and we discuss each in detail.

Figure 15 The Transformer model architecture as first introduced in Vaswani et al. (Reference Vaswani, Shazeer and Parmar2017).

Figure 15Long description

The diagram illustrates two parallel processing branches one on the left beginning with Input and the other on the right starting with Output. In the left branch, an upward arrow leads from Input to Input Embedding, which then connects to a circle with a plus icon. This node also receives input from Positional Encoding. The output of this summation flows into a Multi-Head Attention block, followed by an Add and Norm layer. This is then passed through a Feed Forward network and another Add and Norm layer. On the right side, the Output branch follows a similar structure. An upward arrow leads from Output to Output Embedding, which also connects to a summation node that merges with input from Positional Encoding. This sum flows into the first Multi-Head Attention block, followed by an Add and Norm layer. Then, it passes through a second Multi-Head Attention block and another Add and Norm layer. This is followed by a Feed Forward layer and a final Add and Norm layer. The processed output then moves into a Linear transformation layer, which finally leads to the Output at the top of the right branch.

3.7.1 Encoder

In traditional machine learning models, data is often represented in raw, high-dimensional forms. For transformers, this raw data is transformed into dense, continuous representations known as Input Embeddings. In the context of NLP applications, the input embedding layer transforms discrete tokens – like words or subwords – into dense vectors of a predefined dimension $d_{model}$. These vectors capture semantic relationships and contextual meanings of the tokens, allowing the model to learn complex dependencies within the data. By mapping tokens into a continuous space, embeddings facilitate more efficient and effective learning and processing by the model. For time-series, we can take, for example, 1-D convolutional layer to carry out the embedding step.

Unlike RNNs or CNNs, transformers do not inherently process data in a sequential manner. This poses a challenge for capturing the order of inputs in a sequence. Transformers address this need by employing Positional Encodings, which are combined with the input embeddings to inject positional context into the model. These encodings are designed to be unique for each position in the sequence and can be generated using various methods, such as sinusoidal functions:

$\begin{array}{rl}P{E}_{pos,2i}& =sin(pos/{10000}^{2i/d_{model}}),\\ P{E}_{pos,2i+1}& =cos(pos/{10000}^{2i/d_{model}}),\end{array}$(52)

where $i$ is the dimension and $pos$ is the position. This function is used as we inspect that this simple form allows the model to study the relative position of inputs. Position encodings ensure that the model can distinguish between different positions in the sequence, thereby preserving the order and relational information that is vital for understanding positional information.

Together, input embeddings and position encodings enable transformers to handle sequential data with high flexibility and efficiency. They transform raw features into meaningful representations and incorporate positional information, allowing transformers to model intricate interconnections and dependencies. This combination is a key factor behind the impressive performance of transformers across a variety of tasks in NLP problems and beyond.

The core strength of the Transformer is rooted in the attention mechanism, specifically self-attention, which enables the model to assign varying levels of significance to different elements of the input sequence when encoding each token. Leveraging well-established mathematical foundations, this mechanism effectively manages long-range dependencies. Within the encoder, the Self-Attention Mechanism allows the model to assign varying degrees of importance to different segments of the input sequence for each token. The first step in this process consists of linear projections:

$\begin{array}{rl}{\mathbf{\text{Q}}}_i& ={\mathbf{\text{W}}}^Q{\mathbf{\text{x}}}_i,\\ {\mathbf{\text{K}}}_i& ={\mathbf{\text{W}}}^K{\mathbf{\text{x}}}_i,\\ {\mathbf{\text{V}}}_i& ={\mathbf{\text{W}}}^V{\mathbf{\text{x}}}_i,\end{array}$(53)

where each token’s embedding is transformed into Query ($Q$), Key ($K$), and Value ($V$) vectors via learned weight matrices. Here ${\mathbf{\text{x}}}_i$ represents the internal representation of a single token for NLP tasks or a single timestamp for time-series problems.

Attention scores are determined by the dot product of the Query and Key vectors, scaled by the square root of the Key vector dimension $d_k$ to keep the variance close to 1. These scaled scores are then passed through a softmax function to produce the attention weights:

$\text{attention weights}(x_i,x_j)=\text{softmax}(\frac{{\mathbf{\text{Q}}}_i^T{\mathbf{\text{K}}}_j}{\sqrt{d_k}}),$(54)

where the final representation for each token is computed as a weighted sum of the Value vectors:

${\text{output}}_i=\sum _j\text{attention weights}(x_i,x_j)\cdot {\mathbf{\text{V}}}_j.$(55)

To capture multiple aspects of token relationships, Transformers use multi-head attention. Each head performs self-attention with different sets of weight matrices:

${\text{head}}_h=\text{Attention}({\mathbf{\text{Q}}}^h,{\mathbf{\text{K}}}^h,{\mathbf{\text{V}}}^h),$(56)

where outputs from each attention head are concatenated and then passed through a learned weight matrix:

$\text{Multi Head}=\text{Concat}({\text{head}}_1,\cdots ,{\text{head}}_H){\mathbf{\text{W}}}^O,$(57)

where $H$ indicates the parallel heads. After the self-attention step, each token in the sequence is independently processed by a position-wise feed-forward network, introducing additional nonlinearity and enhancing the transformer’s capacity to capture complex features beyond what self-attention alone can achieve. This network is composed of two linear transformations with a ReLU activation in between. Formally, for each token, this can be represented as:

$\text{FFN}(x)=max(0,x{\mathbf{\text{W}}}_1+{\mathbf{\text{b}}}_1){\mathbf{\text{W}}}_2+{\mathbf{\text{b}}}_2,$(58)

where ${\mathbf{\text{W}}}_1$ and ${\mathbf{\text{W}}}_2$ are trainable weight matrices, ${\mathbf{\text{b}}}_1$ and ${\mathbf{\text{b}}}_2$ are learnable biases, and $max(0,\cdot )$ represents the ReLU activation function. The FFN is applied independently to each position (i.e., each token embedding) and transforms the embeddings into a different feature space. In a two-layer FFN, the first linear transformation is often used to expand the dimensionality, while the second rescales it back to the original size.

Both the self-attention and feed-forward sub-layers incorporate residual connections and layer normalization, which help stabilize training and enhance overall performance. Layer normalization is a technique used to stabilize training by normalizing activations within each training example across the features of a given layer. Through these residual connections, the input to each sub-layer is added directly to its output, and layer normalization is applied to the sum to maintain numerical stability and convergence:

$\text{LayerNorm}(x+\text{Sublayer}(x)).$(59)

Together, these components enable the Transformer encoder to effectively process and encode sequences. By leveraging self-attention to capture dependencies and FFNs to learn complex feature mappings, the Transformer architecture establishes a versatile and powerful foundation for numerous applications.

3.7.2 Decoder

The decoder in a Transformer architecture is integral to generating output sequences for purposes such as machine translation, text generation, and sequence-to-sequence tasks. The decoder processes representations from the encoder and produces sequential output tokens. Its design integrates key elements such as masked self-attention, multi-head attention, and position-wise feed-forward networks, all of which are instrumental in enhancing the decoder’s overall effectiveness.

The decoder’s masked self-attention mechanism enforces that the prediction at any position in the sequence relies only on the previously observed positions, thus maintaining the autoregressive property essential for sequence generation. To achieve this, a mask is applied to the attention weights to block the model from attending to future tokens. At each position $i$ in the decoder, the attention scores are calculated as:

$\text{score}(x_i,x_j)={\mathbf{\text{Q}}}_i^T{\mathbf{\text{K}}}_j,$(60)

where ${\mathbf{\text{Q}}}_i$ and ${\mathbf{\text{K}}}_j$ are the Query and Key vectors, respectively. To prevent the leakage of future information, a mask $M$ is applied to the scores:

$\text{masked score}(x_i,x_j)=\frac{{\mathbf{\text{Q}}}_i^T{\mathbf{\text{K}}}_j}{\sqrt{d_k}}+M_{i,j},$(61)

where $M_{i,j}$ is $-\mathrm{\infty }$ if $j>i$, ensuring that the softmax function will yield zero weights for future tokens:

$\text{attention weights}(x_i,x_j)=\text{softmax}(\text{masked score}(x_i,x_j)),$(62)

where the output for each token $x_i$ is computed as:

${\text{output}}_i=\sum _j\text{attention weights}(x_i,x_j)\cdot {\mathbf{\text{V}}}_j.$(63)

In the decoder, multi-head attention helps the model capture various aspects of the relationships between the decoder’s tokens and the encoder’s output. Through cross-attention, the decoder focuses on the encoder’s output. For each head $h$, the cross-attention mechanism computes:

${\text{head}}_h=\text{Attention}({\mathbf{\text{Q}}}_{dec}^h,{\mathbf{\text{K}}}_{enc}^h,{\mathbf{\text{V}}}_{enc}^h),$(64)

where ${\mathbf{\text{Q}}}_{dec}^h$ are the Query vectors from the decoder, and ${\mathbf{\text{K}}}_{enc}^h$ and ${\mathbf{\text{V}}}_{enc}^h$ are the Key and Value vectors from the encoder. We can concatenate the outputs from each head and transform them as:

$\text{multi head output}=\text{Concat}({\text{head}}_1,\cdots ,{\text{head}}_h){\mathbf{\text{W}}}^o.$(65)

Each decoder position is individually passed through a position-wise feed-forward network to improve its representational ability. This network is generally comprised of two linear layers with a ReLU activation in between. Formally, for each token $x_i$, the transformation is:

$\text{FFN}(x_i)=max(0,x_i{\mathbf{\text{W}}}_1+{\mathbf{\text{b}}}_1){\mathbf{\text{W}}}_2+{\mathbf{\text{b}}}_2,$(66)

where the network enables the decoder to capture complex feature interactions. Residual connections are applied around each sub-layer (self-attention, cross-attention, and feed-forward network), followed by layer normalization. For a given sub-layer output, the layer normalization is:

$\text{LayerNorm}(x+\text{SubLayer}(x)).$(67)

By integrating masked self-attention, multi-head attention, and position-wise feed-forward networks, the decoder’s architecture empowers it to produce coherent and contextually appropriate output sequences. By attending to past and present tokens and incorporating information from the encoder, the decoder can handle long-range input sequences. The final outputs are generated from the decoder autoregressively, meaning that the model relies upon the encoded input sequence and previously generated tokens to produce each subsequent token.

3.7.3 Transformers-Based Time-Series Models

The usage of transformers for time-series analysis has attracted great popularity. Because time-series problems come with their own unique challenges, such as temporal dependencies, irregular sampling, and varying sequence lengths, there has been a surge of research exploring how to tailor transformers specifically for these use cases (see Y. Wang et al. (Reference Wang, Wu and Dong2024) for a recent review). Broadly, one can categorize transformers designed for time-series tasks into two main groups: one based on the application domain, and another based on the underlying network architecture. Figure 16 shows the groupings of these models and their respective sub-fields.

Figure 16 Groups of time-series transformers based on application domains and attention modules.

From an application-domain perspective, time-series transformers are typically tailored toward four major tasks: forecasting, imputation, classification, and anomaly detection. Our main focus here is on forecasting, but we briefly mention all tasks for completeness. Forecasting involves predicting future values based on historical data patterns. Transformers excel here by capturing long-range temporal dependencies that classical models might miss. Imputation deals with filling in missing or corrupted data points. The self-attention mechanism helps the model learn relationships across different time steps to recover lost information. For the classification of sequences, transformers can leverage learned feature embeddings to distinguish between categories or labels associated with entire sequences or specific time intervals. Finally, anomaly detection involves identifying unusual patterns or outliers in the data. Here, transformers can highlight subtle deviations from normal behavior by comparing attention-weighted signals across multiple timestamps.

We can also group models in terms of how attention is employed. Here we distinguish between point-wise, patch-wise, and series-wise. Point-wise attention treats each time step as an individual token, learning direct pairwise relationships across all time steps. This granular approach can capture intricate local patterns, though it might become computationally heavy for very long sequences. Patch-wise attention groups consecutive time steps into patches or segments, reducing the overall sequence length before applying attention. This strategy trades some resolution for improved efficiency and can still preserve local correlations within each patch. Finally, series-wise attention considers the entire sequence as a single unit, using more global operations to learn high-level representations of the data. While this can be extremely efficient, it risks losing some of the fine-grained temporal detail that is critical to many time-series applications. Figure 17 illustrates these attention modules.

Figure 17 Categorization based on attention modules: point-wise, patch-wise, and series-wise. Examples of models that employ each of those attention types respectively are the Informer, PatchTST, and iTransformer.

We list each attention module with one representative work. For point-wise attention, the vanilla transformer and the Informer designed by H. Zhou et al. (Reference Zhou, Zhang and Peng2021) are illustrative examples of earlier transformer models for time-series. A good example for patch-wise attention is PatchTST designed by Nie, Nguyen, Sinthong, and Kalagnanam (Reference Nie, Nguyen, Sinthong and Kalagnanam2022). The series-wise attention can be found in the recently introduced iTransformer (Y. Liu et al. (Reference Liu, Hu and Zhang2023)). There is a trade-off between the benefits from moving from point-wise to sequence-wise attention. Patch-wise strikes a good balance in this trade-off which has led PatchTST to outperform many other architectures in forecasting benchmarks.

Besides, there are many other models that apply one of these attention modules, including the Autoformer (H. Wu et al., Reference Wu, Xu, Wang and Long2021), Crossformer (Y. Zhang & Yan, Reference Zhang and Yan2023), and others. Readers might also find some of the earlier works in this area useful. These are widely covered in an earlier survey paper by Lim and Zohren (Reference Lim and Zohren2021). Note that there are still interesting modules that we have not covered here. For example, the work of Lim, Arık, Loeff, and Pfister (Reference Lim, Arık, Loeff and Pfister2021) on the Temporal Fusion Transformer (TFT) designed a transformer architecture specifically for multi-horizon forecasting, combining the strengths of transformers with recurrent layers to handle both static and time-varying features.

Overall, we think that it is important to recognize the scope and progress that has been made on the development of transformers for time-series applications. It has become clearer which Transformers are best suited for specific time-series challenges, whether that involves capturing nuanced local trends for anomaly detection or learning broad seasonal patterns for long-term forecasting. The interplay between the nature of the data and the chosen architecture continues to shape ongoing innovations in transformer-based time-series modeling. This will continue to pave the way for increasingly accurate and robust models. For interested readers, the aforementioned recent review paper (Y. Wang et al., Reference Wang, Wu and Dong2024) is a good place to start any further reading.

3.8 Graph Neural Networks and Large Language Models

In this section, we introduce some recent developments that have gained great popularity. Note that these materials are more advanced and we include them to demonstrate some directions of future development for applying deep learning models to quantitative finance. In this section, we discuss the intuition of the usage of these methods and introduce various promising resulting applications.

3.8.1 Graph Neural Networks

In the realm of machine learning, the advent of graph neural networks (GNNs) has marked a significant evolution in our ability to process, analyze, and derive insights from data that can be modeled by graphs. Graph representations, with nodes that represent entities and edges that represent their relationships, pervade numerous domains including social networks, molecular chemistry, transportation systems, and communication networks. Traditional neural network models, despite their prowess, fall short when it comes to capturing the dependencies and relational information inherent in graph data. GNNs are a groundbreaking class of neural networks engineered to explicitly handle graph structures and have led to a leap forward in areas such as node classification, link prediction, and graph classification.

In finance, we can often naturally represent the interactions among entities (such as individuals, institutions, and assets) as graphs. GNNs provide a framework to process such graph-structured data. Research on the application of GNNs in quantitative finance has been active in the past few years. The works of Pu, Roberts, Dong, and Zohren (2023); C. Zhang, Pu, Cucuringu, and Dong (Reference Zhang, Pu, Cucuringu and Dong2023) adopt GNNs to build momentum strategies and to forecast multivariate realized volatility. Soleymani and Paquet (Reference Soleymani and Paquet2021); Sun, Wei, and Yang (Reference Sun, Wei and Yang2024) combine GNNs with reinforcement learning to tackle the problem of portfolio construction. For a nice review on GNNs in various financial applications, interested readers are pointed to the work of J. Wang, Zhang, Xiao, and Song (Reference Wang, Zhang, Xiao and Song2021). Here, we introduce the basics of networks and, in particular, we describe the most prevalent GNNs model, graph convolutional neural networks (GCNs).

Basics of Networks and Graphs

The core strength of GNNs stems from their capacity to learn representations of nodes (or entire graphs) that encapsulate not only their features but also the rich context provided by their connections. Such operations are achieved through mechanisms like message passing, aggregating information across neighboring nodes, and iteratively refining their representations. This process allows GNNs to capture both local structures and global graph topology, offering a nuanced understanding of graph-structured data. Before delving into GNNs, we need to understand the basics of networks and graphs. To start, we define a graph $\mathcal{G}$ as:

$\mathcal{G}=(\mathcal{V},\mathcal{E}),$(68)

where $\mathcal{V}=\{v_1,\cdots ,v_n\}$ denotes the set of $n$ nodes and $\mathcal{E}$ represents the set of edges. An edge $e_{ij}=(v_i,v_j)\in \mathcal{E}$ indicates a connection between nodes $v_i$ and  $v_j$.

Nodes (also called vertices) represent the entities or objects in a graph. In different contexts, a node could represent a computer in a network, a person in a social network, a city in a transportation map, or a neuron in a neural network. Edges (also called links) represent the connections or relationships between these nodes. Edges can be undirected, indicating a bidirectional relationship, or directed, indicating a one-way relationship (these form a directed graph or digraph). Edges may also have weights, which quantify the strength or capacity of the connection, such as the distance between cities, bandwidth in a network, or the strength of a social tie.

In order to describe a graph, we need a way to represent nodes and edges in a compact form. This is done in the form of an adjacency matrix. An adjacency matrix A is a $n\times n$ matrix, where A${}_{ij}$ indicates the connectivity status between node $v_i$ and $v_j$. Depending on the nature of the problem, there are many types of graphs:

• Undirected graphs: These are graphs with edges that lack direction, meaning each connection between two nodes is inherently bidirectional.

• Directed Graphs (Digraphs): Graphs in which edges carry a direction, representing a one-way relationship from one node to another. For example, $e_{ij}=(v_i,v_j)\in \mathcal{E}$ denotes an edge pointing from node $v_i$ to node $v_j$.

• Bipartite Graphs: This is a distinct type of graph in which nodes are divided into two separate groups, and every edge connects a node from one group to a node in the other group, with no edges existing within the same group.

• Homogeneous graphs: Graphs where all nodes and edges are of a single type.

• Heterogeneous Graphs: Graphs that contain multiple types of nodes and/or edges. For example, we can denote a graph as $\mathcal{G}=(\mathcal{V},\mathcal{E},t:\mathcal{V}\to \mathcal{A},\tau :\mathcal{E}\to \mathcal{R})$, where each node $v_i\in \mathcal{V}$ is assigned a type $a_i\in \mathcal{A}$ by function $t$ and each edge $e_{ij}\in \mathcal{E}$ is assigned a type $r_{ij}\in \mathcal{R}$ by function $\tau$.

• Dynamic graph: A dynamic graph is defined as a sequence of graphs ${\mathcal{G}}^{seq}=\{{\mathcal{G}}_1,\cdots ,{\mathcal{G}}_T\}$, where each ${\mathcal{G}}_i=({\mathcal{V}}_i,{\mathcal{E}}_i)$ for $i=1,\cdots ,T$. In this sequence, ${\mathcal{V}}_i$, ${\mathcal{E}}_i$ represent the sets of nodes and edges for the $i$-th graph, respectively.

Graph Convolutional Neural Networks

In the work of Z. Wu et al. (Reference Wu, Pan and Chen2020), GNNs are classified into four main categories: convolutional graph neural networks, graph auto-encoders, recurrent graph neural networks, and spatial-temporal graph neural networks. In this section, we introduce graph convolutional neural networks (GCNs) which have become the most widely adopted and extensively utilized GNNs models.

As previously mentioned, CNNs have seen tremendous success in handling data with established grid-like structures, such as images. The fundamental principle of CNNs lies in the convolution operation, which entails moving a filter (or kernel) across the input data (e.g., an image) to generate a feature map. This feature map highlights the presence of particular features or patterns at various positions within the input. This process is inherently suited to data with a regular, grid-like structure where the relative positioning of data points (e.g., pixels in images) is consistent and meaningful.

GCNs (Kipf & Welling, Reference Kipf and Welling2016) extend the concept of convolution to graph-structured data, where the data points (nodes) are connected by edges in a non-Euclidean domain. Unlike the regular, grid-like topology of images, graphs are irregular, and the number of neighbors for each node can be different. GCNs address this by defining convolution in terms of feature aggregation from a node’s neighbors, allowing them to capture the structural information of the graph. The fundamental operation in a GCNs is this aggregation of features from a node’s neighbors, and it can be mathematically represented as:

$H^{l+1}=\sigma ({\hat{D}}^{-\frac{1}{2}}\hat{A}{\hat{D}}^{-\frac{1}{2}}{H}^{(l)}{W}^{(l)}),$(69)

where the matrix of node features at layer $l$ is denoted as $H^{(l)}$ and $H^{(0)}$ is the input feature matrix $X$. We write $\hat{A}=A+I_N$ as the addition between the adjacency matrix $A$ and identity matrix $I_N$, allowing nodes to consider their own features in aggregation. $\hat{D}$ is the degree matrix of $\hat{A}$, where ${\hat{D}}_{ii}=\sum _j{\hat{A}}_{ij}$, and $W^{(l)}$ is the weight matrix for layer $l$. Here $\sigma$ denotes a nonlinear activation function, such as ReLU.

Simply put, much like traditional CNNs are constructed from convolutional layers, GCNs are built by stacking multiple graph convolutional layers. Each graph convolutional layer receives the node vectors from the previous layer (or the initial input feature vectors for the first layer) and generates new output vectors for each node. To illustrate this process, Figure 18 depicts how a graph convolutional layer aggregates the vectors from each node’s neighbors.

Figure 18 A graph convolution layer that pools information for node $A$ from its neighbors.

Figure 18Long description

Four circular nodes: A, B, C, and D. Node A is connected to B and C. Node B is connected to C.Node B is connected to D. Associated with each node (A, B, C, D) is a vertical stack of blue-shaded rectangles labeled X subscript A, X subscript B, X subscript C, and X subscript D. Bottom Left: A mathematical expression for f subscript A is shown, depicting a concatenation of feature vectors X subscript A, X subscript B, X subscript C. An arrow from this concatenation points to a single shaded rectangle labeled h subscript A. Right Side: Three vertical stacks of blue-shaded rectangles representing X subscript A, X subscript B, X subscript C are shown as inputs. These three inputs converge into a single, taller stack of blue-shaded rectangles labeled Aggregate (Average or pool). An arrow (update) from the aggregated feature stack points to a single shaded rectangle labeled h subscript A.

In Figure 18, the vector for node $A$, labeled $x_A$, is combined with the vectors of its neighboring nodes, $x_B$ and $x_C$. This combined vector is then transformed or updated to produce node $A$’s vector in the next layer, denoted as $h_A$. This process is uniformly applied to every node in the graph. This technique is commonly referred to as message passing, where each node “passes” its vector to its neighbors to facilitate the updating of their vectors. The “message” from each node is its associated vector. The specific rules for aggregation and updating are detailed in Equation 69.

Once we stack several graph convolutional layers, we form a typical GCNs as shown in Figure 19. The output of a GCNs depends on the problem at hand. Graph prediction tasks are commonly classified into three categories: graph-level, node-level, and edge-level. When dealing with a node-level task, such as classifying individual nodes, the vectors generated for each node can serve as the final outputs of the model. In the case of node classification, these output vectors may represent the probabilities that each node belongs to specific classes. This is illustrated in the top section of Figure 20.

Figure 19 A GCNs that consists of multilayers.

Figure 19Long description

The multi-layered neural network or graph convolution model. It shows three vertical layers labeled Layer 0 (Input), Layer 1, and Layer 2. Each layer has a similar structure: Left side: A vertical stack of blue-shaded rectangles representing features or embeddings. Middle: A dark gray oval node, connected by solid lines to two light gray oval nodes. Right side: Another vertical stack of blue-shaded rectangles, similar to the left side, representing features. Dashed vertical lines connect the central dark gray node of three layers.

Figure 20 Top: a “node-level” prediction task; Bottom: a “graph-level” prediction task.

Alternatively, we might focus on a “graph-level” task, where the objective is to generate a single output for the entire graph rather than producing outputs for each individual node. For example, the goal could be to classify entire graphs instead of classifying each node separately. In this case, the vectors from all nodes are collectively input into another neural network (such as a simple multilayer perceptron) that processes them together to produce a single output vector. This is illustrated in the bottom part of Figure 20.

Edge-level tasks focus on predicting properties or attributes related to the edges in a graph. These tasks are important for understanding and interpreting the relationships between entities represented by the nodes in a graph. For example, we can predict whether a link (edge) should exist between two nodes, even if it is not present in the observed data. This task is fundamental in various applications, such as recommending friends in social networks, predicting interactions between proteins in biological networks, or inferring missing connections in knowledge graphs.