Developing Prediction Models for COVID-19 Outcomes: A Valuable Tool for Resource-Limited Hospitals

Introduction

Coronavirus disease 2019 (COVID-19), caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), is a global pandemic with millions of confirmed cases and hundreds of thousands of deaths worldwide.¹ While the majority of COVID-19 cases are mild, some individuals experience severe illness requiring hospitalization and intensive care.

Predicting which individuals are at risk for severe COVID-19 can inform the allocation of limited healthcare resources and guide the development of strategies for disease prevention and management. A very important aspect in the development of such prediction models is the differentiation between science and pseudoscience, this type of prediction requires accurate data, reproducibility and specialized researchers involved in their development. Kasapçopur Ö draws attention to this aspect and the need for rigorous documentation of scientific works and the involvement of researchers who have the studied topic as an area of expertise.²

Gao YD et al highlights the importance of clinical and paraclinical factors as risk factors in the evolution of the disease, including age, underlying medical conditions and laboratory results outside the reference range, aspects also showed by Carpenter CR et al.^3,4 It is clear that a varied range of diverse set of indicators can provide predictability regarding the evolution of the disease, including both clinical (disease like obesity, diabetes) and paraclinical factors (leucocyte formula, C-reactive protein). Understanding the role of these factors in the development and progression of COVID-19 can help to identify individuals at high risk for severe illness and guide the implementation of preventive measures.

The goal of the research was to scrutinize easy to obtain available factors, like demographic characteristics of the patient, comorbidities, and blood metrics like white blood count (WBC), absolute neutrophil count (ANC), C-reactive protein (CRP), and utilize them to generate a simple straightforward forecasting model to differentiate whether the disease progression is likely to be favourable or unfavourable. Using this kind of prediction approach empowers resource-constrained hospitals to make timely referrals of their patients to larger, more resource-equipped hospitals.

Materials and Methods

Study Design

The present study is a multivariate observational retrospective study conducted between March 2020 and December 2020 on patients hospitalized in Dr. Victor Babeș Clinical Hospital for Infectious Diseases and Pulmonology Timișoara.

The inclusion criteria for the study were: age above 1 year old (to avoid juxtaposition with neonatal mortality); COVID-19 diagnosis confirmed by polymerase chain reaction (RT-PCR) testing; residency in the municipality of Timișoara, the biggest city in the Western Region of Romania, with 319,279 at the last National census (to reduce the impact of other factors like the quality of primary or emergency medicine in rural areas that could have limited access to immediate medical care prior to hospital admission); availability of data in the medical records or laboratory test results for the parameters considered in the study design. Demographic and clinical data were extracted from electronic medical records (EMR) and included sex, age, comorbidities (medical history) and stage of disease severity. Comorbidities were further categorized as following: cardiovascular diseases – CVD (that included hypertension, ischemic heart disease, arrhythmias, post myocardial infarction status, valvular diseases), chronic pulmonary disease – CPD (that included chronic obstructive bronchopulmonary disease – COPD, bronchial asthma, fibrosis, emphysema), and other comorbidities: obesity, diabetes mellitus, hepatopathy, renal diseases, neurologic disorders, immunodeficiency, autoimmune diseases, cancer.

Disease severity classification comprised 3 stages (mild, moderate and severe) defined in accordance with Protocol no. 487/23.03.2020 and appendix no. 253/27.03.2020. Paraclinical data were extracted from the Hospital’s Laboratory software and included three blood test results: WBC, ANC, CRP results from the first laboratory test performed during hospitalization. The laboratory protocol was the same for patients included in the study. The blood tests were carried out on SYSMEX XN 550 by methods of flow-cytometry, impedance and colorimetry. Reference ranges of blood parameters for normal adults were the following WBC: 4.00–10.00 × 10³/µL, ANC: 2.20–6.60 × 10³/µL, CRP: <5 mg/L. For the paediatric population, the reference values are WBC: 6.20–17 × 10³/µL, ANC: 1.50–7.50 × 10³/µL, CRP: <5mg/L.

Regarding the outcome of the disease, it was divided into two categories in accordance with the final clinical status of the patient extracted from the EMR. A favourable outcome was considered when one of the following two statuses: “discharged - clinically improved” and “discharged - disease free” was present. An unfavourable outcome was linked to the presence of “deceased” status, meaning patient death occurred during hospitalization. No other statuses besides these three were reported.

Patient data was anonymised prior to data extraction, as patients’ CNPs (Personal Numeric Code – the only unique identifying number used in Romania) were replaced with an automatically generated case identification code (procedure done by the hospital staff) and all patient identifiable information (first name, last name and address) were removed.

From a number of 2693 patients admitted with confirmed COVID-19 diagnosis by PCR-testing between March 2020 and December 2020, a total of 483 patients met the inclusion criteria and were included in the present study. The study group was further divided into 3 sub-groups in accordance with the disease severity form.

None of the participants were administered the COVID-19 vaccine as its administration for the general population started in January 2021 in Romania.

The Local Committee of Ethics for Scientific Research of the “Victor Babes” University of Medicine and Pharmacy Timisoara operates under art. provisions 167 of Law no. 95/2006, art. 28, chapter VIII of order 904/2006 and with EU GCP Directives 2005/28/EC, International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH), and with the Declaration of Helsinki—Recommendations Guiding Medical Doctors in Biomedical Research Involving Human Subjects. The current study protocol received ethical approval from the “Victor Babes” University of Medicine and Pharmacy Timisoara (nr. 23/2021).

Statistical Analysis

Statistical analysis was performed using R version 4.2.0 (R Foundation for Statistical Computing, Vienna, Austria).

The R Base Package was used for data manipulation, and the R Stats Package was used for performing univariate analysis (including Pearson’s Chi-squared test, Fisher's Exact Test and Kruskal–Wallis H-test) and multivariate analysis (including Multiple Logistic Regression). The R pROC package was used for analysing and visualizing Receiver Operating Characteristic (ROC) curves, compute their confidence intervals, and perform hypothesis tests.

Continuous variables were summarized as means and standard deviations, while categorical variables were presented as counts and percentages.

For numerical variables, the study utilized non-parametric Kruskal–Wallis H-test to compare multiple independent samples that did not follow a normal distribution, while the chi-square test was used for categorical variables. For situations where frequencies were less than 5, Fisher’s Exact Test was used to evaluate the association between two categorical variables. If these categorical variables consisted of more than two groups, the Freeman-Halton extension of Fisher’s Exact Test was implemented. The severity of COVID-19 was evaluated using Spearman Correlation and univariate logistic analysis and included clinical and paraclinical collected data elements as risk factors. In addition, risk factors were further analysed using multiple logistic regression to develop diagnostic models for distinguishing between a favourable outcome (discharge) and an unfavourable outcome (death) of COVID-19 cases. The clinical and paraclinical features’ ability to distinguish between the two types of COVID-19 was determined using Receiver Operating Characteristic (ROC) analysis, which found the corresponding cut-off value.

To assess the relationship between dependent variable and predictor variables, we used statistical test(s) with a significance level of 0.05.

Imputation of missing data was not used during statistical analysis.

Results

A total of 483 patients with COVID-19 met the inclusion criteria and were included in the study group for analysis.

Considering disease severity form at the moment of initial evaluation as the main predictive factor of the disease evolution, the study group was further divided into 3 subgroups: mild form subgroup (39.33%, N=190), moderate form subgroup (36.23%, N=175), severe form subgroup (24.43%, N=118). Clinical characteristics of the study group and severity forms subgroups are presented in Table 1.

Table 1 Distribution of Demographic, Clinical Characteristics, and Comorbidities of Patients Across the Different Severity Forms Subgroups

Prediction Factors and Disease Severity Forms

Age

The mean age of the study group was 61.19 years (range: min 9 y.o, max 95 y.o., standard deviation SD=14.15). The mean age for the 3 subgroups increased as disease severity increased: for the mild group the mean age was 56.85 (range: min 9 y.o, max 95 y.o., standard deviation SD=15.00), for the moderate group it was 61.44 (range: min 26 y.o, max 92 y.o., standard deviation SD=12.65), for the severe group it was 67.83 (range: min 36 y.o, max 92 y.o., standard deviation SD=12.19).

Sex

Males represented 56.11% (N=271) of the study group, while females represented 43.89% (N=212). Significant differences between the 3 subgroups were observed regarding sex distribution, as males were more frequently affected by moderate (sex distribution for the moderate group: male/female = 60%/40%, p<0.001) and severe forms (sex distribution for the severe group: male/female= 66.95/33.05%, p<0.001). Females were more frequent in the mild group (sex distribution for the mild group: male/female = 45.79%/54.21%).

Comorbidities

Cardiovascular pathology, obesity (all forms) and renal disease all varied with statistical significance between the three subgroups; in contrast, diabetes mellitus, hepatopathy, neurological disorder, immunodeficiency, autoimmune disease, and cancer did not vary between the subgroups.

In the severe subgroup, 76.27% (N=90) presented cardiovascular diseases, 26.27% (N=31) had diabetes mellitus and 16.95% (N=20) suffered from renal disease (Table 1).

Paraclinical Factors

All the three studied paraclinical factors (WBC, ANC, CRP) varied significantly between the subgroups (Table 2). The mean CRP value for the severe group was 114.99, almost 23 times higher than the upper limit of the normal range.

Table 2 Comparison of Paraclinical Factors Values Across the Different Severity Forms Subgroups

Prediction Factors and Disease Outcome

Disease outcome varied with high statistical significance between the three subgroups (Table 3). The unfavourable outcome was present in 5.27% (N=10) of the cases in the mild subgroup, 9.71% (N=17) in the moderate subgroup, and 71.79% (N=72) in the severe group (p<0.001).

Table 3 Disease Outcome in the Study Group and Severity Forms Subgroups

The Spearman correlation and univariate logistic analysis of clinical risk factors for COVID-19 unfavourable outcome is presented in Table 4. The results indicate that age is strongly correlated with an unfavourable outcome (p<0.001). This means that there is a statistically significant difference in the mean age of individuals who died from COVID-19 in the study group compared to those who did not.

Table 4 Association Between Various Demographic Characteristics, Disease Severity Form, Comorbidities, and Unfavourable Outcome in the Study Group, Based on Spearman Correlation and Univariate Logistic Regression Analysis

In contrast, sex does not appear to be significantly correlated with an unfavourable outcome, as indicated by a p-value of 0.741 in the Pearson’s Chi-squared test. Similarly, chronic pulmonary disease (CPD), obesity, diabetes, hepatopathy, immunodeficiency, autoimmune disease and cancer do not appear to be significantly correlated with unfavourable. However, cardiovascular disease (CVD), renal diseases and neurological disorder are all significantly correlated with poor outcome, as indicated by p-values of <0.001. Having a CVD increases the chance of death 3.37 times.

The Spearman correlation and univariate logistic analysis of paraclinical factors for COVID-19 unfavourable outcome is presented in Table 5. Increased values of WBC, ANC and CRP are all correlated with an unfavourable outcome.

Table 5 Correlation Between Paraclinical Factors and Unfavourable Outcome in the Study Group, Based on Spearman Correlation and Univariate Logistic Regression Analysis

In the multiple logistic regression model for clinical factors (Table 6), the unfavourable outcome was positively associated with age (aOR=1.06, 95% CI 1.03–1.09, p<0.001), severe form (aOR=1.81, 95% CI 1.00–34.08, p<0.001) and neurological disorders (aOR=2.91, 95% CI 1.10–7.64, p<0.05). The model had good fit, as indicated by low deviance residuals, the low p-value of the coefficients and the comparison of AIC (Akaike Information Criterion) value with other considered models.

Table 6 Correlation Between Selected Clinical Risk Factors and Unfavourable Outcome in the Study Group Using Multivariate Logistic Regression

In the multiple logistic regression model for paraclinical factors (Table 7), the unfavourable outcome was positively associated with ANC (aOR=1.53, 95% CI 1.10–2.33, p<0.05) and CRP (aOR=1.004, 95% CI 1.001–1.008, p<0.005). The model had good fit, as indicated by a low deviance residuals and p-value of the coefficients.

Table 7 Correlation Between Paraclinical Risk Factors and Unfavourable Outcome in the Study Group Using Multivariate Logistic Regression

Prediction Models

Using the three numerical factors that showed high statistical correlation with unfavourable outcome in the multiple logistic regressions (age, ANC, CRP), three diagnostic binary classification models were proposed to separate high-risk and low-risk patients. The ROC curves for the three logistic models in the study group (presented in Figure 1) were Model 1: logit(p) = −2.221925 + 0.006981 * X (X = CRP value); ROC curve: AUC = 0.688 (95% CI: 0. 0.625–0.751, p<0.001); cut-off = 0.13; sensitivity = 72.3%; specificity = 58.8%. Model 2: logit(p) = −2.988112 + 0.005666 * X1 + 0.121688 * X2 (X1 = CRP value, X2 = ANC value); ROC curve: AUC = 0.731 (95% CI: 0.673–0.790, p<0.001); cut-off = 0.14; sensitivity = 71.0%; specificity = 65.7%. Model 3: logit (p) = −8.647672 + 0.003347 * X1 + 0.122706 * X2 + 0.088323 * X3 (X1 = CRP value, X2 = ANC value, X3 = Age); ROC curve: AUC = 0.824 (95% CI: 0.770–0.877, p<0.001); cut-off value = 0.14; sensitivity = 82.8%; specificity = 69.9%.

Figure 1 Receiver operating characteristic (ROC) curves illustrating the performance of proposed predictive models in distinguishing high-risk and low-risk patients within the study group dataset.

As the three included parameters (age, ANC, CRP) were available at the moment of diagnosis in a total of 1186 patients with COVID-19, models and ROC curve were reiterated for this extended group.

The ROC curves for the three logistic models in the extended group (presented in Figure 2) were as follows: Model 1: logit(p) = −2.5072060 + 0.0076854 * X (X = CRP value); ROC curve: AUC = 0.737 (95% CI: 0.698–0.776, p<0.001); cut-off = 0.11; sensitivity = 72.9%; specificity = 64.9%. Model 2: logit (P) = −3.393918 + 0.005682 * X1 + 0.146949 * X2 (X1 = CRP values, X2 = ANC value); ROC curve: AUC = 0.780 (95% CI: 0.743–0.818, p<0.001); cut-off = 0.10; sensitivity = 64.7%; specificity = 79.5%. Model 3: logit (P) = −8.295374 + 0.003499 * X1 + 0.132701 * X2 + 0.079695 * X3 (X1 = CRP value, X2 = ANC value, X3 = Age); ROC curve: AUC = 0.845 (95% CI: 0.813–0.877, p<0.001); cut-off value = 0.12; sensitivity = 72.3%; specificity = 83.9%.

Figure 2 Receiver operating characteristic (ROC) curves illustrating the performance of proposed predictive models in distinguishing high-risk and low-risk patients within the extended group dataset.

When the extended group was used in the models’ creation, the accuracy of the results slightly increased. This stands as proof that the usability of the proposed model brings value in a clinical setup at initial presentation and independently of the disease severity form (a classification that was not available for the extended group and was not used in the model).

Discussion

In respect of age as a risk factor, the mean age was 61.19 years for the study group and 68.83 for the severely diseased group. In contrast, a study from the same medical centre published by Laza et al in 2022 reported a mean age of 48.76 years in the general group.⁵ This important difference could have been caused by the different residency of included patients, as the Laza et al study group also included hospitalized patients from all neighbouring counties, from both rural and urban areas. Because all cases from these administrative divisions were sent to a tertiary centre like the one where the study was performed, patients were admitted independently of age and severity, thus generating a lower mean age. Our results indicate a strong correlation between age and unfavourable outcome, establishing age as an independent risk factor for COVID 19 poor outcome. Laza et al objectivated the same result.⁵ Ho FK et al, 2020 came to the same conclusion, proposing an attenuated immune response among the elderly as the possible explanation.⁶ A systematic review by Romero et al, 2021 that included studies from variate geographic regions (Europe, USA) found the same correlation between increased disease severity and higher age.⁷ Flaherty et al, in a critical literature review from 2020 also reported age as a risk factor, patients over 66 years of age developed more frequent severe forms requiring admission to ICU.⁸

Regarding comorbidities, CVD, renal disease, and neurological disorder were all significantly correlated with an unfavourable outcome in the study group. Our findings show that having a neurologic disorder increases the risk of an unfavourable outcome 5.46 times, a renal disease increases the risk 4.26 times, and CVD 3.37 times.

Garcia et al, in a cohort study from 2020 including 576 patients, found that the 105 patients (18.3% of the total) with neurological disorders prior to COVID-19 infection had a survival rate that was 30 days lower compared with the group of patients without neurological pathology. Patients with chronic neurological disorders developed severe forms of the disease and having a neurological disorder prior to COVID-19 infection led to an unfavourable outcome.⁹ According to Alberti et al, (2021) patients diagnosed with a cerebrovascular disease prior to COVID-19 infection presented a greater vulnerability to secondary events and to an unfavourable outcome.¹⁰

Our study showed that the presence of a kidney disease prior to confirmation of COVID-19 infection increased the risk of an unfavourable outcome 4 times. The same conclusion was reported by Pecly et al in a meta-analysis from 2021 including 73 studies that aimed to evaluate the association between multi-organ dysfunction and the evolution of the disease in patients with COVID-19.¹¹ Pakhchanian et al, in a multicentre cohort study from 2021, that included a total number of 152,463 patients and 8810 patients presenting renal pathology prior to COVID-19 infection, also concluded that renal disease is an independent risk factor for COVID-19 unfavourable outcome.¹²

Li et al, in a study carried out in Wuhan China on a study group containing 596 patients hospitalized between January and March 2020, from which 215 (36.1%) presented CVD prior to COVID-19 infection, also highlighted CVD as an independent risk factor for an unfavourable outcome.¹³ Cordero et al, in a meta-analysis from 2020 that included a number of 307,596 patients with COVID-19 from which 46,321 (15.1% of the total) had CVD, demonstrated that CVD clearly augments the severity of COVID-19, increasing 4 times the risk of an unfavourable outcome.¹⁴

Concerning the contribution of paraclinical parameters, our results show that the unfavourable outcome was positively associated with increased values for ANC and CRP. Bouayed et al, in a monocentric descriptive retrospective study conducted in 2022, correlated the existence of elevated values for CRP in patients with COVID-19 with the severity of the disease and proved that increased values of this marker are a significant risk factor for an unfavourable outcome of the disease.¹⁵ The same conclusion was reported by Lentner et al in 2021 in a retrospective cohort study performed on 541 patients, mentions the presence of high values for CRP at admission as a predictive factor for unfavourable disease evolution.¹⁶

Mosquera‐Sulbaran et al, 2021 studied the role of increased CRP value as a prediction risk factor for COVID-19 unfavourable outcome and showed association with severe lung damage.¹⁷ Mocanu et al, in a retrospective assessment from 2022 with the aim of determining the characteristics of Roma patients with severe forms of the disease, also established the increased level of CRP as a risk factor for the severity of the disease.¹⁸

Huang et al, 2020, in a meta-analysis that included 1289 cases of COVID-19, reported the presence of leucocytosis in patients with severe forms of disease and an unfavourable outcome.¹⁹ This finding was also reported by Yamada et al in a systematic review from 2020, which concluded that leucocytosis and elevated CRP values represented predictive factors for an unfavourable outcome.²⁰ A retrospective study conducted by Zhu et al in 2021 on 163 patients reported a significant association between the presence of leucocytosis at admission and severe forms of the disease that had an unfavourable outcome.²¹

In our study, the increased level of ANC was correlated with a severe form of the disease and an unfavourable outcome. McKenna et al, in a review article from 2022, showed that an increased number of neutrophils in patients with COVID-19 indicated an exacerbation of the disease and was correlated with an unfavourable outcome.²² Moreover, in a 2021 study conducted in Delhi, on a group of 203 patients diagnosed with COVID-19, Agarwal et al reported neutrophilia as a risk factor for a severe evolution of the disease and an unfavourable outcome.²³

Our intention was to establish a model that could be useful to select different management patterns for patients at admission based on outcome prediction: favourable or unfavourable. In our multivariate logistic model, when all three paraclinical factors were considered, just ANC and CRP proved to be independent risk factors. Therefore, after adding age (also being a numerical independent risk factor), we have created several predicting models that use these 3 easy-to-collect parameters: age, ANC and CRP. The best model obtained an AUC of 0.845, sensitivity of 72.3% and specificity of 83.9%, which are on par with other models in the literature.

In a study by Hu et al from 2020 a severe COVID-19 risk model was also built using multivariate logistic regression analysis adjusted by age, lobular involvement score and lymphocyte cell count, showing that an objective score based on medical imaging could also be accustomed.²⁴ Huang et al, who used the same parameters to create the prediction model, reached a higher AUC of 0.903 and a higher sensitivity rate of 90.9%.²⁵ Even using CT score alone can generate a sensitivity of 80.0% and a specificity of 82.8% in the ROC curve interpretation, as shown by Li et al.²⁶ Nevertheless, computer tomography is an investigation that might not be available in remote clinical setups.

A systematic review published by Miller et al in 2022 attempted to assess existing research on prediction models for serious COVID-19 disease that can be used in the emergency room or public health scenarios.²⁷ Knight et al created a prediction model that might be used to divide patients admitted with COVID-19 into various care groups. The recommended easy-to-use 4C Mortality Score included 8 variables that could be measured at initial hospital presentation, one of them being CRP. This calculated score proved to be superior to the existing models, being valuable when including patients in various risk groups for an unfavourable outcome and establishing the treatment behaviour appropriate for the case.²⁸ The PAWNN score proposed by Liu et al only uses age and some complete blood count (CBC) results to precisely forecast mortality for COVID-19 patients throughout their entire hospital. It achieved an AUROC of 0.97 in a validation cohort of 2949 patients with only 1 CBC record, demonstrating its role of helping medical professionals prioritize treatment for COVID-19 patients.²⁹ Lopez et al found that distinctly adding the rate of changes of both hemogram ratios (neutrophil-to-platelet-ratio and neutrophil-to-lymphocyte-ratio) during the first week after admission slightly improves AUC and prediction power of the proposed models.³⁰

Nevertheless, the majority of these studies solely investigated in-hospital mortality as the outcome of interest, a common limitation also presented in our study.^27–30

This research has several constraints. Firstly, the study was limited by the relatively small number of participants. Secondly, the study was based on data from a single center, limiting the generalizability of the findings. Furthermore, the study did not incorporate frequently used parameters in ICU settings, such as oxygen saturation levels, clotting cascade activation, and supportive drug treatments, which could have provided further insights into patient outcomes and disease progression.

Other studies have also achieved similar findings, underscoring the need for a straightforward and swift predictive model that could be implemented to differentiate between patient’s outcomes. This would enable timely and appropriate treatment for those predicted to have an unfavourable outcome, ideally in specialized medical units equipped with resources tailored to their needs.

Conclusion

The present study showed that advanced age, the presence of comorbidities (CVD, renal disease, neurological pathology), and increased values for WBC, ANC and CRP represent important risk factors for an unfavourable COVID-19 outcome. Thus, a composite profile that includes older age, certain comorbidities, leucocytosis, neutrophilia, and elevated CRP values might suggest an unfavourable prognosis. Conversely, a younger age, absence of comorbidities, and normal CBC and CRP values might signify a more favourable outcome.

To aid in predicting outcomes, we proposed a prediction model for an unfavourable outcome based on very accessible parameters such as age, ANC and CRP value. The model characterizes COVID-19 in the capital city of the Western Region of Romania, Timișoara.

Building on the results of our study, this model could be particularly beneficial for decision-making in healthcare, such as whether to admit patients in smaller hospitals or to directly transfer those potentially facing an unfavourable outcome to larger hospitals equipped with more extensive care options. In this way, the model could assist healthcare practitioners in optimizing patient management and treatment, ultimately improving COVID-19 outcomes.