Performance of Three Measures of Comorbidity in Predicting Critical COVID-19: A Retrospective Analysis of 4607 Hospitalized Patients

Introduction

During 2020, the rapid spread of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and the severity of coronavirus disease 2019 (COVID-19) led to the collapse of many healthcare systems worldwide, particularly hospital and intensive care unit (ICU) resources. Recently approved vaccines against SARS-CoV-2 infection are expected to ease hospital resources pressure.¹ Nevertheless, in many settings, COVID-19 continues to cause high hospital demand, which requires adequate healthcare provision planning, particularly in scenarios of a shortage of resources or overburdening of hospitals.

Early in the pandemic, underlying comorbidities were pointed out as significant prognostic factors for the development of severe illness. Initial analyses of a large series of COVID-19 patients revealed that hospitalized patients with chronic conditions like diabetes, hypertension, chronic obstructive pulmonary disease, or cardiovascular diseases were more likely to develop severe COVID-19, with no consensus regarding the contribution of each comorbidity to explaining differences in COVID-19 outcomes.^2–5 Based on this evidence, the patient’s comorbidity burden on admission has been included in many models for predicting hospital outcomes in COVID-19 patients, sometimes combined with post-admission information (eg, vital signs, laboratory and imaging results).⁶ However, the comorbidity burden is typically represented by a simple count of few chronic conditions (ie, up to 12) that does not fully capture patients’ complexity.^7–14

Another approach to representing the patient’s comorbidity burden is using a comorbidity index that combines information about several comorbidities into a single score.^17,18 This approach yields more complex models with a more challenging interpretation; however, it allows adjusting for multiple factors, likely improving model accuracy.¹⁸ Known examples of comorbidity indices include the Charlson Comorbidity Index¹⁵ and the Elixhauser index for hospitalized patients.¹⁶ In 2020, we developed a comprehensive risk index tool for hospitalized patients, the Queralt Index, which includes a measure of pre-existing comorbidities (Queralt DxS).¹⁹ The Queralt DxS combines and weighs more than 2100 relevant acute and chronic diagnostic codes. Queralt DxS has shown optimal performance for risk adjustment when measuring comorbidity burden in hospitalized patients.¹⁹

Understanding the risk factors, such as comorbidities associated with hospital outcomes in COVID-19 patients, is crucial for healthcare planning during future waves of the infection. However, most predicting models published to date rely on simple counts of comorbidities, and no studies have explored the appropriate measure for adjusting for multimorbidity in the COVID-19 setting. In this study, we assessed the performance of Queralt DxS as a comorbidity measure in models for predicting outcomes in hospitalized COVID-19 patients and compared it with other widely used comorbidity measures: the Charlson and Elixhauser indices.

Materials and Methods

Data Sources

In this retrospective analysis, we included data from patients hospitalized due to COVID-19 in any of the eight hospitals of the Catalan Institute of Health (ICS), the leading health care provider in Catalonia (North-East Spain; 7.5 million inhabitants). The ICS provides universal health care to nearly 70% of the Catalan population, and accounts for approximately 30% of all hospitalizations reported in Catalonia. ICS hospitals systematically collect and store data on diagnoses and resource utilization into a centralized database. We screened the ICS database for all admissions due to COVID-19 occurred within the investigated period, which was restricted to June 15–December 8, 2020, to prevent biases associated with the unprecedented hospital overburdening experienced in Spain during the first wave of the COVID-19 outbreak.^20,21 The database was locked on March 30, 2021.

Hospital admissions due to COVID-19 were identified according to the following codes of the International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM) system: B97.29, B97.21, B34.2, J12.81, J12.89, and the recently added code for lab-confirmed COVID-19 U07.1. We included all hospital admissions of patients who either died in the hospital or were discharged home. Records from patients transferred from or to other hospitals were excluded from the analysis to ensure the completeness of clinical data during the full index hospitalization.

All data were handled according to the General Data Protection Regulation 2016/679 on data protection and privacy for all individuals within the European Union and the local regulatory framework regarding data protection. The independent ethics committee of the Bellvitge Biomedical Research Institute (IDIBELL) approved the study protocol (Ref. PR195/21) and waived the need for informed consent, as the data were generated as part of routine clinical care and fully de-identified for analytic purposes.

Outcomes and Predictors

The study outcome (“critical illness” in patients hospitalized with COVID-19) was a composite that included the need for invasive mechanical ventilation, transfer to the intensive care unit (ICU), or in-hospital death. The need for invasive mechanical ventilation was identified by the presence of any of the following procedure codes recorded in the hospital database for billing purposes: 5A09357, 5A09457, 5A09557, 5A1935Z, 5A1945Z, 5A1955Z, 09HN7BZ, 09HN8BZ, 0BH13EZ, 0BH17EZ, 0BH18EZ, and 0CHY7BZ.

Potential predictors considered for the analysis included age, sex, and comorbidities present on admission. The latter were measured using three multi-comorbidity indices: the Charlson Comorbidity Index, the Elixhauser index, and Queralt DxS. In another model, we also used the 27 comorbidities included in the Elixhauser index, each of them separately. The Charlson and Elixhauser indices were estimated using the ICD-10 coding system proposed by Quan et al.²² For the Charlson index, diagnostic weights were assigned based on the original formulation by Charlson et al.¹⁵ For the Elixhauser index, we used the weights proposed by Moore et al.²³ Queralt DxS belongs to a family of three indices for predicting clinical outcomes in hospitalized patients that provide a numerical value from the weighted sum of diagnoses (primary, secondary present on admission, and complications) from a list of 2119 diagnostic code groups.¹⁹ In this analysis, we used an updated version of the index (version 6.0), computed using healthcare data recorded in the routine care setting between 2018 and 2019.

Statistics

Data on demographic and clinical variables and outcomes were described as frequency and percentage, mean and standard deviation (SD), and/or median and interquartile range (IQR, defined as the 25th and 75th percentiles), as appropriate. The Charlson and Elixhauser indices were computed using the Comorbidity library by Gasparini;²⁴ Queralt DxS was generated using R functions and instructions provided in the Supplementary File. For descriptive purposes only, patients were also stratified at four risk levels (ie, low, moderate, high, and very high) based on the 50th, 80th, and 95th percentiles of Queralt DxS, as previously described.¹⁹

Since the primary objective was to compare the performance of each index of comorbidity when added to a given multivariate risk prediction model, we built five logistic regression models to predict the composite outcome of critical illness: a baseline model with age (introduced into the model as a continuous variable) and sex, three models including the baseline and each of the comorbidity measures (ie, Charlson index, Elixhauser index, and Queralt DxS), and one including the 27 diagnoses included in the Elixhauser index separately. The performance of each model was evaluated using five statistical measures of model performance: the deviance, the Akaike information criterion (AIC),²⁵ the Bayesian Information Criterion (BIC),²⁶ the area under the receiver operating characteristic curve (AUROCC), and the area under the precision–recall curve (AUPRC).²⁷ The 95% confidence intervals (CI) of AUROCC were estimated using 1000 bootstrap samples and confirmed with the DeLong criteria.²⁸ All analyses were performed using the R statistical package (version 4.0.3).²⁹

Results

Study Population

Between June 15 and December 8, 2020, 4607 patients were admitted to the ICS hospitals with COVID-19. Table 1 summarizes the demographic characteristics of the study population according to the incidence of critical illness (ie, need for invasive mechanical ventilation, transfer to ICU, or death) on admission and/or during hospitalization. The corresponding comorbidity burden, measured using the three analyzed indices, is shown in Table 2. Patients who experienced critical illness on/after admission were significantly older and had higher scores of comorbidity burden irrespective of the index used. Men experienced critical illness more frequently.

Table 1 Demographic Characteristics of Patients Included in the Analysis, Grouped by Incidence of Critical Illness During the Index Hospitalizationa

Table 2 Clinical Characteristics of Patients Included in the Analysis, Grouped by Incidence of Critical Illness During the Index Hospitalizationa

According to the distribution of patients across the five risk levels of Queralt DxS, 2145 (46.6%) patients were at low risk, 1509 (32.8%) at moderate risk, 730 (15.8%) at high risk, and 223 (4.8%) at very high risk. Figure 1 shows the distribution of patients with and without critical illness, according to age, sex, and Queralt risk group.

Figure 1 Participant distribution according to age, sex, and Queralt risk group (corresponding to the 50th, 80th, and 95th percentiles of the Queralt Index).

Performance of Comorbidity Indices for Prediction of Critical COVID-19 Illness

Table 3 shows the overall performance of each model, computed using various measures of the magnitude of the effect. The addition of a multimorbidity measure to the baseline model increased the AUROCC and AUPRC values of the model, which were consistent with all other metrics of model performance. In all, the best performance was consistently observed for the model that included age, sex, and Queralt DxS (Table 3). Figure 2 shows the ROC and precision–recall curves of the five models explored.

Table 3 Performance of the Baseline Model and Models Including Each of the Investigated Measures of Comorbidity

Figure 2 Performance of the five logistic regression models: baseline (age and sex), Charlson (age, sex, and Charlson index), Elixhauser (age, sex, and Elixhauser index), Queralt (age, sex, and Queralt DxS index), and the 27 diagnostic codes included in the Elixhauser index. (A) receiver operating characteristics curve. (B) precision–recall curve.

In the three models including a comorbidity index, this factor significantly contributed to predicting critical illness, irrespective of the comorbidity index used in the model (Figure 3). In the model including Queralt DxS as a comorbidity measure, age groups lost statistical significance.

Figure 3 Standardized coefficients according to four logistic regression models: baseline (age and sex), Charlson (age, sex, and Charlson index), Elixhauser (age, sex, and Elixhauser index), and Queralt (age, sex, and Queralt DxS).

Discussion

Our results using data from the healthcare databases of Catalonia confirm the relevance of comorbidities on admission as a strong predictor of outcomes in patients hospitalized with COVID-19, which is consistent with prior reports. Accordingly, the addition of any of the summary measures of comorbidity burden used in our analysis improved the performance of predictive models compared with age and sex only. Furthermore, for the first time to our knowledge, we showed how different measures of the comorbidity burden can change the accuracy of models for predicting critical illness in hospitalized COVID-19 patients.

In the context of comorbidity burden being a strong predictor of outcomes in COVID-19 patients, we observed that the most comprehensive multi-comorbidity index evaluated, Queralt DxS, showed the strongest contribution to explaining critical illness. Unlike the other two indices investigated (ie, the Charlson and Elixhauser indices), which estimate comorbidity burden from a discrete list of diagnoses, Queralt DxS considers all conditions present on admission among a list of 2119 diagnostic groups. Hence, our findings align with the concerns highlighted by Ording and Sørensen, who warned about residual confounding potentially introduced when underestimating the comorbidity burden by using numerical indices based on rather restrictive definitions of comorbidity.¹⁸ Remarkably, when introducing an exhaustive summary measure of comorbidities, such as Queralt DxS, to the model, age was no longer a significant predictor. This suggests that the risk of critical illness in older patients is driven by the accumulation of chronic conditions over time rather than by age per se.

In addition to comorbidities, various authors have identified other relevant predictors of hospital outcomes in patients with COVID-19, such as inflammatory biomarkers, need for oxygen therapy, and diagnostic images analyzed using advanced machine learning approaches.^30–35 While these models are helpful for informing clinical decisions based on assessments performed during the hospital stay in individual patients, models that use retrospective information available from electronic health records can provide hospital outcome estimates at the time of hospital admission, thus aiding not only clinicians but also managers and policymakers in hospital resource planning. Our results suggest that all models, irrespective of whether they include data from in-hospital assessments, may benefit from a summary, highly comprehensive measure of the comorbidity burden on admission, such as Queralt DxS. We have made this index and the related code freely available online for research purposes as an open-source tool to facilitate evaluation by other groups either in its current form in other populations or potentially as part of clinical risk estimation tools combining comorbidities and clinical data on admission.

Indices that summarize a high number of variables into a single numerical index, such as Queralt DxS, also have the advantage of allowing more parsimonious models. This is particularly important for machine learning approaches, which may lose performance when increasing the number of variables included in the model. Hence, despite losing sight on the individual effect of each diagnosis, summarizing the information from thousands of pre-existing diagnoses into a single index that accounts for the relative weights of each one and their prognostic relevance for hospital outcomes may enrich other models while minimizing the risk of overfitting.

Study Limitations

One of the limitations of Queralt DxS is the lack of validation in settings other than Catalonia; however, we hope that making the software publicly available for research purposes will facilitate external validation moving forward. Moreover, it is worth mentioning that the weights used to estimate the relative contribution of each diagnostic group to health risk were calculated using data collected between 2018 and 2019, before the COVID-19 pandemic. Hence, although the source population of the eight relevant hospitals has remained unchanged since then, the sample of patients admitted with COVID-19 included in this study, and their clinical and sociodemographic profiles do differ from those used when developing the Queralt indices, including Queralt DxS. Therefore, this analysis may be interpreted as a pseudo-validation of Queralt DxS in a Catalan subpopulation hospitalized with COVID-19.

Also, as often occurs in retrospective analyses, our dataset was limited to the data recorded in electronic records during routine care. Nonetheless, the universal coverage of our healthcare system, and the cross-linking of healthcare data from the primary care and specialized settings allowed us to access various sources and very comprehensive health data from our study population and consider all possible diagnoses present on admission, as well as basic demographic data, such as age and sex. Conversely, we were not able to include some pre-admission variables such as blood groups, which have been associated with COVID-19 severity³⁶ and might modulate the weight of comorbidities when estimating the risk of critical illness, or the SARS-CoV-2 variant, which may influence clinical outcomes of COVID-19 patients. Nevertheless, even without those pieces of information, the performance of Queralt DxS was very robust.

Finally, it is worth mentioning that we did not adjust for treatments administered or the type of care received. COVID-19 is an evolving pandemic, and the development and implementation of pathogenesis-directed therapies³⁷ and care pathways³⁸ may strongly influence the clinical outcome, regardless of the baseline risk factors of hospitalized patients. Due to the relatively narrow investigated period and early stage of the pandemic (ie, June–December 2020), we expect the type of care to be homogeneous among study patients. However, future analyses are warranted to investigate how newly developed therapies, vaccination, and improved care pathways influence the weight of the comorbidity burden on predicting critical illness.

Conclusion

Our findings show that the burden of pre-existing comorbidities significantly improves the prediction of the risk of critical illness in patients hospitalized with COVID-19, particularly when measured exhaustively using a tool such as Queralt DxS. This comorbidity index, which is freely available for research purposes, may improve the accuracy of risk models aimed at supporting clinical decision-making and hospital resource planning in hospitalized patients, including those with COVID-19. Future studies investigating the influence of the comorbidity burden, measured with the Queralt DxS tool, on hospital outcomes in the context of vaccination and improved care pathways are warranted.