Using Principal Component Analysis (PCA) Techniques to Improve Multiple Regression Performance in High-Dimensional Data
Abstract
High-dimensional data represent one of the major challenges in statistical modeling, as the inflation in the number of predictors often leads to multicollinearity and instability in the estimates of multiple regression coefficients. Principal Component Analysis (PCA) is considered an effective statistical tool to address this issue by transforming correlated variables into a set of uncorrelated components while retaining the maximum possible variance in the data. This study aims to evaluate the role of PCA in improving the performance of multiple regression in high-dimensional settings, by applying it to synthetic simulated data (300–600 observations, 50–300 predictors) and to real-world data (Coffee Quality Dataset, 1,339 observations, 43 variables). The results of Principal Component Regression (PCR) were compared with Ordinary Least Squares (OLS) and regularized models such as Ridge, Lasso, and Elastic Net. The models were evaluated using common statistical measures, including the Root Mean Square Error (RMSE), the Mean Absolute Error (MAE), and the coefficient of determination (R²), in addition to cross-validation. The findings revealed that applying PCA prior to regression effectively reduced the impact of multicollinearity and improved the accuracy and stability of the models, emphasizing PCA as a practical and efficient approach for handling high-dimensional data and enhancing predictive modeling.
Downloads
References
[1] Pearson, K. (1901). LIII. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11), 559–572. https://doi.org/10.1080/14786440109462720
[2] Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24, 417–441, 498–520. https://doi.org/10.1037/h0071325
[3] Jackson, J. E. (1991). A User’s Guide to Principal Components. Wiley, New York.https://DOI:10.1002/0471725331
[4] Jolliffe, I. T., Trendafilov, N. T., & Uddin, M. (2003). A modified principal component technique based on the LASSO. Journal of computational and Graphical Statistics, 12(3), 531-547. https://doi.org/10.1198/1061860032148
[5] McCabe, G. P. (1984). Principal Variables. Technometrics, 26(2), 137–144. https://doi.org/10.1080/00401706.1984.10487939
[6] Ringnér, M. (2008). What is principal component analysis? Nature Biotechnology, 26, 303–304. https://doi.org/10.1038/nbt0308-303
[7] Johnstone, I. M., & Lu, A. Y. (2009). On consistency and sparsity for principal components analysis in high dimensions. Journal of the American Statistical Association, 104, 682–693. https://doi.org/10.1198/jasa.2009.0121
[8] Audigier, V., Husson, F., & Josse, J. (2015). Multiple imputation for continuous variables using a Bayesian principal component analysis. Statistics and Computing, 25, 727–747. http://arxiv.org/abs/1401.5747
[9] Jung, S. (2018). Continuum directions for supervised dimension reduction. Computational Statistics & Data Analysis, 120, 112–125. http://arxiv.org/abs/1606.05988
[10] Kaplan, A., & Lock, E. F. (2017). Prediction With Dimension Reduction of Multiple Molecular Data Sources for Patient Survival. Cancer informatics, 16, 1176935117718517. https://doi.org/10.1177/1176935117718517
[11] Tianyi Zhou, Dacheng Tao, Xindong Wu (2010). Manifold Elastic Net: A unified framework for sparse dimension reduction. Machine Learning, 81, 79–113. http://arxiv.org/abs/1007.3564
[12] Bersimis, S., Psarakis, S., & Panaretos, J. (2007). Multivariate statistical process control charts: an overview. Quality and Reliability engineering international, 23(5), 517-543. https://doi.org/10.1002/qre.829
[13] Fan, Y., & Fan, Z. (2023). A time series regression model via improved PCA and bagging algorithms. Francis Academic Press. Retrieved from https://francis-press.com/papers/10223
[14] Guerra-Urzola R, Van Deun K, Vera JC, Sijtsma K. A Guide for Sparse PCA: Model Comparison and Applications. Psychometrika. 2021 Dec;86(4):893-919. doi: 10.1007/s11336-021-09773-2. Epub 2021 Jun 29. PMID: 34185214; PMCID: PMC8636462.
[15] Greenacre, M., Groenen, P. J. F., Hastie, T., D’Enza, A. I., Markos, A., & Tuzhilina, E. (2022). Principal component analysis. Nature Reviews Methods Primers, 2, 100. https://doi.org/10.1038/s43586-022-00184-w
[16] He, Y., Wang, G., & Yang, Y. (2024). Sparse Principal Component Analysis with Non-Oblivious Adversarial Perturbations. arXiv preprint arXiv:2411.05332. arXiv preprint arXiv:2411.05332
[17] Teresa, N., Hogg, D. W., & Villar, S. (2022). Dimensionality reduction, regularization, and generalization in overparameterized regressions. SIAM Journal on Mathematics of Data Science, 4(1), 126-152. https://doi.org/10.1137/20M1387821
[18] Huang T, Li J, Zhang W. Application of principal component analysis and logistic regression model in lupus nephritis patients with clinical hypothyroidism. BMC Med Res Methodol. 2020 May 1;20(1):99. doi: 10.1186/s12874-020-00989-x. PMID: 32357838; PMCID: PMC7195728.
[19] Lukman, A. F., Adewuyi, E. T., Alqasem, O. A., Arashi, M., & Ayinde, K. (2024). Enhanced Model Predictions through Principal Components and Average Least Squares-Centered Penalized Regression. Symmetry, 16(4), 469. https://doi.org/10.3390/sym16040469
[20] Yan, Q., Yang, C., & Wan, Z. (2023). A Comparative Regression Analysis between Principal Component and Partial Least Squares Methods for Flight Load Calculation. Applied Sciences, 13(14), 8428. https://doi.org/10.3390/app13148428
[21] Ziwei Zhu, Tengyao Wang, Richard J. Samworth, )2022.( High-Dimensional Principal Component Analysis with Heterogeneous Missingness, Journal of the Royal Statistical Society Series B: Statistical Methodology, Volume 84, Issue 5, Pages 2000–2031, https://doi.org/10.1111/rssb.12550
[22] Zhou, T., Tao, D., & Wu, X. (2020). Manifold elastic net: A unified framework for sparse dimension reduction. Machine Learning, 81, 79–113. https://doi.org/10.1007/s10994-010-5206-3
[23] Green , A., & Romanow, E. (2025). The high dimensional asymptotics of principal component regression .The Annals of Statistics, 53(4), 1697 -1725 https://doi.org/10.1214/25-Aos2532
[24] Wu,Y.F., Zhu,Y.,Cao,L.,& Shi,N.(2025).Calibrated Principal Component Regression. arXiv https://arxiv.org/abs/2510.19020
Published
Deprecated: Return type of Carbon\Traits\Date::createFromTimestamp($timestamp, $tz = null) should either be compatible with DateTime::createFromTimestamp(int|float $timestamp): static, or the #[\ReturnTypeWillChange] attribute should be used to temporarily suppress the notice in /home/iqserver/journals/kjeas.uowasit.edu.iq/lib/pkp/lib/vendor/nesbot/carbon/src/Carbon/Traits/Timestamp.php on line 29
2026-07-09
