Hostname: page-component-745bb68f8f-s22k5 Total loading time: 0 Render date: 2025-01-10T22:21:15.077Z Has data issue: false hasContentIssue false
  • English
  • Français

FROM MODEL SELECTION TO MODEL AVERAGING: A COMPARISON FOR NESTED LINEAR MODELS

Published online by Cambridge University Press:  07 January 2025

Wenchao Xu Open the ORCID record for Wenchao Xu [Opens in a new window]  and
Xinyu Zhang
Wenchao Xu
Affiliation:
Shanghai University of International Business and Economics
Xinyu Zhang*
Affiliation:
Academy of Mathematics and Systems Science, Chinese Academy of Sciences
*
Address correspondence to Xinyu Zhang, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China; e-mail: xinyu@amss.ac.cn
Get access Rights & Permissions [Opens in a new window]

Abstract

Model selection (MS) and model averaging (MA) are two popular approaches when many candidate models exist. Theoretically, the estimation risk of an oracle MA is not larger than that of an oracle MS because the former is more flexible, but a foundational issue is this: Does MA offer a substantial improvement over MS? Recently, seminal work by Peng and Yang (2022) has answered this question under nested models with linear orthonormal series expansion. In the current paper, we further respond to this question under linear nested regression models. A more general nested framework, heteroscedastic and autocorrelated random errors, and sparse coefficients are allowed in the current paper, giving a scenario that is more common in practice. A remarkable implication is that MS can be significantly improved by MA under certain conditions. In addition, we further compare MA techniques with different weight sets. Simulation studies illustrate the theoretical findings in a variety of settings.

Type
ARTICLES
Information
Econometric Theory , First View , pp. 1 - 33
Copyright
© The Author(s), 2025. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

We thank the Editor (Peter C.B. Phillips), the Co-Editor (Liangjun Su), two anonymous referees, Jingfu Peng, Yundong Tu, and Yuhong Yang for many constructive comments and suggestions. Xu’s research was partially supported by the National Natural Science Foundation of China (12101591). Zhang’s research was partially supported by the National Natural Science Foundation of China (71925007, 72091212, and 71988101), Beijing Natural Science Foundation (Z240004), and the CAS Project for Young Scientists in Basic Research (YSBR-008).

References

REFERENCES

Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In Petroc, B., & Csake, F. (Eds.), Second international symposium on information theory (pp. 268–281). Akademiai Kiado.Google Scholar
Alhorn, K., Dette, H., & Schorning, K. (2021). Optimal designs for model averaging in non-nested models. Sankhya A , 83, 745–778.CrossRefGoogle Scholar
Alhorn, K., Schorning, K., & Dette, H. (2019). Optimal designs for frequentist model averaging. Biometrika , 106, 665–682.CrossRefGoogle ScholarPubMed
Allen, D. M. (1974). The relationship between variable selection and data agumentation and a method for prediction. Technometrics , 16, 125–127.CrossRefGoogle Scholar
Ando, T., & Li, K.-C. (2014). A model-averaging approach for high-dimensional regression. Journal of the American Statistical Association , 109, 254–265.CrossRefGoogle Scholar
Ando, T., & Li, K.-C. (2017). A weight-relaxed model averaging approach for high-dimensional generalized linear models. The Annals of Statistics , 45, 2654–2679.CrossRefGoogle Scholar
Andrews, D. W. (1991). Asymptotic optimality of generalized CL , cross-validation, and generalized cross-validation in regression with heteroskedastic errors. Journal of Econometrics , 47, 359–377.CrossRefGoogle Scholar
Claeskens, G., Croux, C., & Van Kerckhoven, J. (2006). Variable selection for logistic regression using a prediction-focused information criterion. Biometrics , 62, 972–979.CrossRefGoogle ScholarPubMed
Ding, J., Tarokh, V., & Yang, Y. (2018). Model selection techniques: An overview. IEEE Signal Processing Magazine , 35, 16–34.CrossRefGoogle Scholar
Fang, F., Lan, W., Tong, J., & Shao, J. (2019). Model averaging for prediction with fragmentary data. Journal of Business & Economic Statistics , 37, 517–527.CrossRefGoogle Scholar
Fang, F., & Liu, M. (2020). Limit of the optimal weight in least squares model averaging with non-nested models. Economics Letters , 196, 109586.CrossRefGoogle Scholar
Feng, Y., & Liu, Q. (2020). Nested model averaging on solution path for high-dimensional linear regression. Stat , 9, e317.CrossRefGoogle Scholar
Feng, Y., Liu, Q., Yao, Q., & Zhao, G. (2022). Model averaging for nonlinear regression models. Journal of Business & Economic Statistics , 40, 785–798.CrossRefGoogle Scholar
Hansen, B. E. (2007). Least squares model averaging. Econometrica , 75, 1175–1189.CrossRefGoogle Scholar
Hansen, B. E. (2014). Model averaging, asymptotic risk, and regressor groups. Quantitative Economics , 5, 495–530.CrossRefGoogle Scholar
Hansen, B. E., & Racine, J. S. (2012). Jackknife model averaging. Journal of Econometrics , 167, 38–46.CrossRefGoogle Scholar
He, B., Liu, Y., Wu, Y., Yin, G., & Zhao, X. (2020). Functional martingale residual process for high-dimensional Cox regression with model averaging. Journal of Machine Learning Research , 21, 1–37.Google Scholar
Hjort, N. L., & Claeskens, G. (2003). Frequentist model average estimators. Journal of the American Statistical Association , 98, 879–899.CrossRefGoogle Scholar
Ing, C.-K. (2007). Accumulated prediction errors, information criteria and optimal forecasting for autoregressive time series. The Annals of Statistics , 35, 1238–1277.CrossRefGoogle Scholar
Ing, C.-K., & Wei, C.-Z. (2005). Order selection for same-realization predictions in autoregressive processes. The Annals of Statistics , 33, 2423–2474.CrossRefGoogle Scholar
Lehrer, S., & Xie, T. (2017). Box office buzz: Does social media data steal the show from model uncertainty when forecasting for Hollywood? Review of Economics and Statistics , 99, 749–755.CrossRefGoogle Scholar
Li, K.-C. (1987). Asymptotic optimality for Cp ,CL , cross-validation and generalized cross-validation: Discrete index set. The Annals of Statistics , 15, 958–975.CrossRefGoogle Scholar
Liao, J., Zou, G., Gao, Y., & Zhang, X. (2021). Model averaging prediction for time series models with a diverging number of parameters. Journal of Econometrics , 223, 190–221.CrossRefGoogle Scholar
Liu, C.-A. (2015). Distribution theory of the least squares averaging estimator. Journal of Econometrics , 186, 142–159.CrossRefGoogle Scholar
Liu, Q., & Okui, R. (2013). Heteroscedasticity-robust ${C}_p$ model averaging. The Econometrics Journal , 16, 463–472.CrossRefGoogle Scholar
Liu, Q., Okui, R., & Yoshimura, A. (2016). Generalized least squares model averaging. Econometric Reviews , 35, 1692–1752.CrossRefGoogle Scholar
Magnus, J. R., & Luca, G. D. (2016). Weighted-average least squares (WALS): A survey. Journal of Economic Surveys , 30, 117–148.CrossRefGoogle Scholar
Magnus, J. R., Powell, O., & Prüfer, P. (2010). A comparison of two model averaging techniques with an application to growth empirics. Journal of Econometrics , 154, 139–153.CrossRefGoogle Scholar
Mallows, C. (1973). Some comments on Cp . Technometrics , 15, 661–675.Google Scholar
Moral-Benito, E. (2015). Model averaging in economics: An overview. Journal of Economic Surveys , 29, 46–75.CrossRefGoogle Scholar
Peng, J., & Yang, Y. (2022). On improvability of model selection by model averaging. Journal of Econometrics , 229, 246–262.CrossRefGoogle Scholar
Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics , 6, 461–464.CrossRefGoogle Scholar
Shao, J. (1997). An asymptotic theory for linear model selection. Statistica Sinica , 7, 221–242.Google Scholar
Shibata, R. (1983). Asymptotic mean efficiency of a selection of regression variables. Annals of the Institute of Statistical Mathematics , 35, 415–423.CrossRefGoogle Scholar
Steel, M. (2020). Model averaging and its use in economics. Journal of Economic Literature , 58, 644–719.CrossRefGoogle Scholar
Stone, M. (1974). Cross-validation choice and assessment of statistical procedures. Journal of the Royal Statistical Society: Series B , 36, 111–147.CrossRefGoogle Scholar
Trench, W. F. (1999). Asymptotic distribution of the spectra of a class of generalized Kac-Murdock-Szegö matrices. Linear Algebra and its Applications , 294, 181–192.CrossRefGoogle Scholar
Wan, A. T., Zhang, X., & Zou, G. (2010). Least squares model averaging by Mallows criterion. Journal of Econometrics , 156, 277–283.CrossRefGoogle Scholar
Wang, M., Zhang, X., Wan, A. T., You, K., & Zou, G. (2023). Jackknife model averaging for high-dimensional quantile regression. Biometrics , 79, 178–189.CrossRefGoogle ScholarPubMed
Yan, X., Wang, H., Wang, W., Xie, J., Ren, Y., & Wang, X. (2021). Optimal model averaging forecasting in high-dimensional survival analysis. International Journal of Forecasting , 37, 1147–1155.CrossRefGoogle Scholar
Yang, Y. (1999). Model selection for nonparametric regression. Statistica Sinica , 9, 475–499.Google Scholar
Yuan, Z., & Yang, Y. (2005). Combining linear regression models: When and how? Journal of the American Statistical Association , 100, 1202–1214.CrossRefGoogle Scholar
Zhang, X. (2021). A new study on asymptotic optimality of least squares model averaging. Econometric Theory , 37, 388–407.CrossRefGoogle Scholar
Zhang, X., Ullah, A., & Zhao, S. (2016a). On the dominance of Mallows model averaging estimator over ordinary least squares estimator. Economics Letters , 142, 69–73.CrossRefGoogle Scholar
Zhang, X., Wan, A. T., & Zou, G. (2013). Model averaging by jackknife criterion in models with dependent data. Journal of Econometrics , 174, 82–94.CrossRefGoogle Scholar
Zhang, X., & Wang, W. (2019). Optimal model averaging estimation for partially linear models. Statistica Sinica , 29, 693–718.Google Scholar
Zhang, X., Yu, D., Zou, G., & Liang, H. (2016b). Optimal model averaging estimation for generalized linear models and generalized linear mixed-effects models. Journal of the American Statistical Association , 111, 1775–1790.CrossRefGoogle Scholar
Zhang, X., Zou, G., Liang, H., & Carroll, R. J. (2020). Parsimonious model averaging with a diverging number of parameters. Journal of the American Statistical Association , 115, 972–984.CrossRefGoogle ScholarPubMed
Zhao, S., Zhou, J., & Li, H. (2016). Model averaging with high-dimensional dependent data. Economics Letters , 148, 68–71.CrossRefGoogle Scholar
Supplementary material: File

Xu and Zhang supplementary material

Xu and Zhang supplementary material
Download Xu and Zhang supplementary material(File)
File 346.2 KB