Performance of Comprehensive Risk Adjustment for the Prediction of In-Hospital Events Using Administrative Healthcare Data: The Queralt Indices

Introduction

Accurate risk adjustment is crucial in fields such as healthcare management, benchmarking and research. This has direct implications for the evaluation of health policies and interventions, the allocation of healthcare resources, and healthcare quality.^1,2 In the current epidemiological era of non-communicable chronic diseases becoming pandemic,³ accurate risk adjustment has become particularly challenging, with a large proportion of hospitalized patients being elderly. Patients now often have multiple concurrent conditions,⁴ and frequently develop complications during a hospital stay.^5,6

Comorbidity indices such as Charlson’s⁷ or Elixhauser’s⁸ have been widely used as standard methods for risk adjustment in a number of settings.^1,9,10 Importantly, although “comorbidities” and “risk” are often used as synonym terms, these actually refer to different concepts, the former being only one of the components of the latter.¹¹ In this context, comorbidity-only indices such as Charlson’s or Elixhauser’s may fail to fully capture a patient’s risk, and more comprehensive tools may be needed.

We have recently developed the Queralt Indices, a family of risk measurements for hospitalized patients, which combine and weigh more than 2100 relevant acute and chronic diagnosis codes, as compared to Charlson’s index, which includes 17 pre-specified chronic diseases, or Elixhauser’s, which originally combined 30. The Queralt Indices allow integrating information from pre-existing comorbidities, in-hospital complications and principal discharge diagnoses.

The aim of the present study was to compare the performance of classic comorbidity-based functions such as Charlson’s and Elixhauser’s, of a more comprehensive comorbidity index (Queralt’s Index for secondary hospital discharge diagnoses, which excludes in-hospital complications; “Queralt DxS”), of the comprehensive Queralt’s Index for any hospital discharge diagnoses (which includes pre-existing comorbidities, in-hospital complications, and principal discharge diagnosis; “Queralt Dx”), and of the All Patients Refined Diagnosis Related Groups (APR-DRG), which also capture information on medical procedures. We compared these tools in terms of their ability to predict key clinical outcomes in hospitalized patients.

Materials and Methods

Setting and Data Source

The present study was conducted in Catalonia (Spain), a Mediterranean, European region with more than 7 million inhabitants, 81% of whom live in urban municipalities. The Catalan Institute of Health (ICS) is the largest public healthcare provider and serves 75% of those insured in Catalonia.

For the present analysis we used the ICS population-based, administrative healthcare database. This database captures information on medical diagnoses generated in 8 Catalan public hospitals. Medical diagnoses are coded in the database using the International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM) system.¹²

Study Design and Study Population

This was an observational, retrospective cohort study. The unit of analysis was hospital discharge, and we included all hospital discharges (either home or dead) registered in the ICS database between January 1st and December 31st, 2018. Only standard hospital stays were included in the analyses (ie, hospital stays for major ambulatory surgical interventions and other procedures were excluded). A same patient could contribute several hospital discharges during the study period.

Computation of the Queralt Indices

Information on medical diagnoses (including preexisting comorbidities, primary discharge diagnosis, and in-hospital complications [defined as any non-principal discharge diagnosis not present on admission]), was obtained from hospital discharge reports using definitions based on ICD-10-CM. In the ICS database, secondary discharge diagnoses not present on admission are flagged as in-hospital complications, as compared to secondary discharge diagnoses already present on admission (i.e., preexisting comorbidities).

The Queralt family of indices currently includes three simple measures for principal discharge diagnosis (Queralt DxP), pre-existing comorbidities (Queralt DxS), and in-hospital complications (Queralt DxC), respectively, and a comprehensive index combining all of them (Queralt Dx). Specifically, the development of the Queralt DxS index was based on a highly computational statistical algorithm, consisting of two differentiated parts: 1) identification of significant diagnosis codes groups, and 2) weighing the effect of each of them. The number of significant diagnosis codes groups identified was 2572, of which 2119 obtained a non-zero weight.

In this report we present the results for the Queralt DxS and Queralt Dx indices, as likely the currently two most relevant measures within the Queralt family.

Computation of Other Measures

As comorbidity-only measures, we calculated Charlson’s⁷ and Elixhauser’s⁸ indices, using all the information available on secondary diagnoses coded in the hospital discharge reports. Charlson’s index combines and weighs a pre-specified set of 17 chronic conditions present on admission,⁷ while Elixhauser’s uses a set of 30 comorbidities defined as conditions present on admission and unrelated to the principal discharge diagnosis. Specifically, we used Quan’s version of both indices,¹³ and van Walraven’s update to calculate the weights in Elixhauser’s measure.¹⁴

As case-mix measures, we computed the APR-DRG Severity level, and the APR-DRG Risk level (version 35).¹⁵

Study Outcomes

Three outcomes of interest were defined: in-hospital death, occurrence of a hospital stay lasting >14 days, and need for a stay in the intensive care unit (ICU). In-hospital deaths (from any cause) and stays in the ICU are identified automatically and recorded exhaustively in the ICS database, for all patients and hospital stays. Length of hospital stay was calculated (in days) based on the hospital discharge and hospital admission dates, both of which are recorded systematically in the database. The cut-point of 14 days (as a proxy of a long in-hospital stay) was defined based on the 90th percentile of the distribution of length of hospital stays in our area.

Statistical Analyses

The unit of analysis was hospital discharge. Demographic characteristics and the frequency of recorded conditions were described using number and proportion (%). We also described the number and % of hospital discharges in which each of the three study outcomes were present.

To understand the number of hospitalizations in which each comorbidity index might provide useful information for risk adjustment purposes, we calculated and plotted the proportion of hospitalizations in which no discharge diagnosis codes were considered relevant by a given index according to Charlson’s, Elixhauser’s and Queralt DxS indices, respectively. Results were presented stratified by age.

To compare the different measures in terms of their discriminative ability, we built logistic regression models for each of the three study outcomes as dependent variables, using each of the indices as independent variables. Models were adjusted for age, sex, type of admission (unplanned, planned), and center, as follows:

• Model 0 (basic model): Age, Sex, Admission Type, Center
• Model 1: Model 0, Charlson’s
• Model 2: Model 0, Elixhauser’s
• Model 3: Model 0, APR-DRG Severity level
• Model 4: Model 0, APR-DRG Risk level
• Model 5: Model 0, Queralt DxS
• Model 6 (Queralt Dx): Model 0, Queralt DxP, Queralt DxS, Queralt DxC

For each of these regression models, we computed the following measures of discrimination and goodness of fit: the area under the receiver operating characteristic (ROC) curve (AUC),¹⁶ Akaike’s information criterion (AIC),¹⁷ the Bayesian information criterion (BIC),¹⁸ and Brier’s score.¹⁹ For the ROC analyses, we computed 95% confidence intervals (CIs) using De Long’s method,²⁰ and plotted the ROC curves of each of the indices for each of the study outcomes.

Subgroup analyses were also conducted, assessing the performance of these measures specifically in individuals with a principal discharge diagnosis of selected types of cancer (colon, pancreatic, lung cancer, leukemias), cardiovascular disease (CVD; acute myocardial infarction, cardiac arrhythmias, heart failure, acute cerebrovascular disease), and respiratory conditions (pneumonia, influenza, acute bronchitis, chronic obstructive pulmonary disease and bronchiectasis). Conditions were grouped using the Clinical Classifications Software coding system, which groups ICD-10 codes.²¹ Additional subgroup analyses were conducted among young (0 to 29 years) and older (≥60 years) patients, as well as by type of admission (planned, unplanned). Also, a sensitivity analysis excluding delivery-related hospitalizations was conducted.

In order to prevent overfitting by age, specifically for APR-DRG we also built regression models excluding age as covariate, as follows:

• Model 7: Sex + Admission Type + Center + APR-DRG Severity level
• Model 8: Sex + Admission Type + Center + APR-DRG Risk level

All analyses were conducted using R software, version 3.5.3.²² R’s statistical package Comorbidity was used for the calculation of Charlson’s and Elixhauser’s indices.²³

Results

Study Population and Characteristics

We included 156,459 hospital discharges occurring between January 1st and December 31st, 2018 (Table 1). The majority of patients were men (51.6%), and most were 60 years of age or older (53.5%). The vast majority of hospital admissions were unplanned (67.1%), and heart failure was the most frequent principal discharge diagnosis (N=5008) among the conditions evaluated.

Table 1 Number and Characteristics of the Hospital Discharges Included in the Study

Study Outcomes

Table 1 also presents the frequency of each of the three study outcomes. Overall, 4.3% hospitalizations resulted in an in-hospital death, 7.5% lasted more than 14 days, and 5.3% included an ICU stay. Men had higher in-hospital mortality than women, their hospital stays were longer, and needed ICU stays more often. In-hospital mortality increased with age, with a high of 9.8% for patients 80 years or older; conversely, long hospital stays were more frequent in newborns. Unplanned stays were associated with higher in-hospital mortality and with longer stays. The highest in-hospital mortality was observed in patients with respiratory failure as their principal diagnosis, followed by patients with cerebrovascular disease and patients with cancer.

Relevant Hospital Discharge Diagnoses by Comorbidity Index

Figure 1 displays the number of hospitalizations in which no discharge diagnoses were considered relevant for risk-adjustment purposes according to Charlson’s, Elixhauser’s and Queralt DxS indices, respectively. The number of these hospitalizations was very high using Charlson’s and Elixhauser’s, especially at young ages. Conversely, this number was much lower using the Queralt DxS index, particularly in younger patients, and was almost zero in elderly individuals.

Figure 1 Proportion of hospital discharges with no relevant diagnoses for risk-adjustment purposes according to each comorbidity index, stratified by age.

Prediction of in-Hospital Death

Among comorbidity-only measures, Elixhauser’s index showed a better performance for the prediction of in-hospital death, in terms of AUC, than Charlson’s. Nevertheless, the Queralt DxS index showed the best discriminative ability among the 3 measures (Table 2, and Figure 2).

Table 2 In-Hospital Death: AUC Analysis for Comorbidity-Only Measures

Figure 2 ROC curves for the prediction of in-hospital death, all hospital discharges. Queralt Dx includes principal diagnosis, pre-existing comorbidities, and in-hospital complications; Queralt DxS includes pre-existing comorbidities. Abbreviations: APR-DRG, All Patients Refined Diagnosis Related Groups; ICU, intensive care unit; ROC, receiver operating characteristic.

When both comorbidity-only and more comprehensive risk adjustment measures were considered, the Queralt Dx index was superior to the APR-DRG, and both outperformed comorbidity-only measures (Tables 2 and 3, and Figure 2).

Table 3 In-Hospital Death: AUC Analysis for APR-DRG Measures and the Queralt Dx Index

The same trends were true in goodness-of-fit analyses in terms of AIC (Table S1), BIC (Table S2), and Brier’s score (Table S3), with the Queralt DxS consistently showing the best performance among comorbidity-based measures, and the Queralt Dx showing the best goodness-of-fit for the prediction of in-hospital death when all measures were considered.

Consistent findings were also observed in AUC subgroup analyses by discharge diagnosis –patients with a primary discharge diagnosis of selected types of cancer, CVD, and respiratory conditions, respectively (Tables 2 and 3). Nevertheless, 95% CIs were wider than in the overall analysis.

Prediction of Length of Hospital Stay >14 Days

As for in-hospital death, in AUC analyses for the prediction of long hospital stay Elixhauser’s index showed a better performance than Charlson’s. Nonetheless, the Queralt DxS index showed the best discriminative ability among the three comorbidity-only measures (Table 4, and Figure 3).

Table 4 Length of In-Hospital Stay >14 Days: AUC Analysis for Comorbidity-Only Measures

Figure 3 ROC curves for the prediction of length of stay >14 days, all hospital discharges. Queralt Dx includes principal diagnosis, pre-existing comorbidities, and in-hospital complications; Queralt DxS includes pre-existing comorbidities. Abbreviations: APR-DRG, All Patients Refined Diagnosis Related Groups; ICU, intensive care unit; ROC, receiver operating characteristic.

When both comorbidity-only and more comprehensive risk adjustment tools were considered, the Queralt Dx index showed once again the best discriminative ability, with both Queralt Dx and APR-DRG outperforming that of comorbidity-only measures (Tables 4 and 5, and Figure 3). The same was true in terms of goodness-of-fit analyses, including AIC (Table S4), BIC (Table S5) and Brier’s score (Table S6) calculations.

Table 5 Length of In-Hospital Stay >14 Days: AUC Analysis for APR-DRG Measures and the Queralt Dx Index

Similar trends were observed in AUC analyses by principal discharge diagnosis (Tables 4 and 5), although 95% CIs were once again wider than in the overall analysis. The only exception to this was observed in the subgroup of patients with a principal discharge diagnosis of leukemia, in which Charlson’s and Elixhauser’s indices showed a better performance than the Queralt DxS. Nonetheless, in these patients Queralt Dx also showed the best performance.

Prediction of ICU Stay

As for the other study outcomes, in AUC analyses the Queralt DxS showed once again a better discriminative ability for the prediction of ICU stay than Charlson’s and Elixhauser’s indices (Table 6), and the Queralt Dx showed the best performance across all measures (Tables 6 and 7, and Figure 4). The same was true in AIC (Table S7), BIC (Table S8) and Brier’s score (Table S9) goodness-of-fit analyses.

Table 6 ICU Stay: AUC Analysis for Comorbidity-Only Measures

Table 7 ICU Stay: AUC Analysis for APR-DRG Measures and the Queralt Dx Index

Figure 4 ROC curves for the prediction of ICU stay, all hospital discharges. Queralt Dx includes principal diagnosis, pre-existing comorbidities, and in-hospital complications; Queralt DxS includes pre-existing comorbidities. Abbreviations: APR-DRG, All Patients Refined Diagnosis Related Groups; ICU, intensive care unit; ROC, receiver operating characteristic.

Similar trends were observed in AUC analyses by principal discharge diagnosis (Tables 6 and 7). The exceptions to this were observed in the subgroups of patients with a principal discharge diagnosis of lung cancer and acute myocardial infarction, in which the performance of Queralt DxS was similar to that of the basic model. Nevertheless, in these patients Queralt Dx improved the performance of Charlson’s and Elixhauser’s indices, and had a similar predictive ability as models using APR-DRG.

Consistent results were also observed in subgroup analyses by age and by type of hospitalization, as well as in a sensitivity analysis excluding delivery-related hospitalizations (Table S10).

Discussion

In a population-based analysis including 156,459 hospital discharges from the general population of Catalonia occurring during 2018, a comprehensive comorbidity index (the Queralt DxS) provided relevant risk adjustment information in a larger number of patients compared to the more parsimonious Charlson’s and Elixhauser’s functions, in all age groups. Consistent with this, the Queralt DxS index outperformed the former for the prediction of in-hospital death, long in-hospital stay, and need for ICU stay in hospitalized patients. In addition, incorporation of information on principal discharge diagnoses and in-hospital complications (the Queralt Dx index) outperformed Charlson’s and Elixhauser’s indices even further, and yielded superior overall predictive ability compared to the APR-DRG. The results of several subgroup and sensitivity analyses were consistent with the overall study findings, highlighting the applicability of the Queralt Indices to different patient subgroups and types of hospitalization. The present findings have important implications for improving current risk-adjustment paradigms.

Of note, the Queralt Indices are intended to be a comprehensive family of indices to be used for risk adjustment purposes, rather than for accurate prediction of specific outcomes. For this reason, while it would have been appropiate to exclude, for example, planned hospitalizations when developing a model intended to predict the risk of in-hospital death among urgent hospitalizations, because our aim was to develop and evaluate a risk adjustment measure, we included all types of hospitalization. Consistent with this aim, the models adjusting for each of these risk adjustment tools were compared in terms of their discrimination ability (AUC), rather than their callibration. In the models, key potential confounders (age and type of hospitalization) were adjusted for. Also, to address concerns for potential bias resulting from the inclusion of delivery-related hospitalizations, a sensitivity analysis excluding these was conducted. The results were consistent with those of the main analysis.

In our study, Elixhauser’s outperformed Charlson’s index for the prediction of the 3 study outcomes. This is consistent with prior studies,^24,25 and is likely the consequence of the former being a more comprehensive measure of chronic comorbidities than the latter. Our results would support prioritizing Elixhauser’s over Charlson’s index for risk adjustment purposes.

The Queralt DxS index provided relevant risk adjustment information in a much larger number of patients than Charlson’s and Elixhauser’s indices. This was particularly true in younger patients, and is likely explained by the larger number of diagnoses considered relevant by Queralt functions – more than 2100, including both acute and chronic processes – compared to the other indices, which combine very few, exclusively chronic conditions. This likely limits their ability to fully capture a patient’s comorbidity burden. Moreover, the Queralt DxS showed improved predictive ability compared to Charlson’s and Elixhauser’s, which have been two of the gold-standards of comorbidity-based risk adjustment for years.^26,27 Therefore, the Queralt DxS index may allow for a more granular comorbidity-based risk adjustment, even within relevant disease groups. This may be relevant for example in patients with heart failure, who may be very heterogeneous in terms of their risk of events.²⁸

In addition, we also observed that inclusion, as part of the risk adjustment strategy, of information on principal discharge diagnoses and on in-hospital complications (the Queralt Dx index) improved discriminative ability and goodness-of-fit even further. This yielded results that outperformed those from the Queralt DxS, and were superior to those from gold-standard case-mix measures such as the APR-DRG. These findings provide support to the notion that risk adjustment should not only consider pre-existing comorbidities, but go beyond those, and incorporate these other features, which also impact a patient’s risk of events.¹¹

The present findings have important implications for current risk adjustment paradigms. In fields in which such adjustment is done retrospectively (e.g., benchmarking), comprehensive comorbidity indices and even more comprehensive tools capturing the whole in-hospital clinical process (i.e., also including principal discharge diagnosis and in-hospital complications) provide an invaluable opportunity to improve current standards beyond strategies based on parsimonious, comorbidity-only indices. In the era of electronic medical records and of widespread use of automated disease coding systems in hospital discharge reports, comprehensive risk adjustment measures such as Queralt’s can be computed easily and automatically, using readily available information. This has the potential to inform healthcare managers and researchers more accurately.

In terms of clinical risk prediction, because the Queralt Dx index uses information generated during the hospitalization, it would be expected to have limited utility for risk assessment on admission. Nonetheless, and although this is not the main purpose of these tools, the Queralt DxS index, which is solely based on pre-existing comorbidities, appears as a potentially valuable tool to identify patients at higher odds of dying, having a long in-hospital stay, or needing an ICU stay during the index hospitalization; more accurately than with traditional tools such as Charlson’s or Elixhauser’s indices. Moreover, it is possible that risk estimations based on comprehensive information generated during preceding hospitalizations (e.g., the Queralt Dx index) could have a value in clinical risk prediction in subsequent in-hospital stays. Nonetheless, further studies are needed to test this hypothesis.

Study Limitations

Some limitations are worth discussing. First, we used an administrative healthcare database to develop the Queralt indices, as well as to conduct the present comparative analysis. Administrative databases are known to have intrinsic limitations when used for research purposes, including the potential for under-recording of medical conditions, or the limited validity of some diagnosis codes, among others.^29,30 Nonetheless, in the last decade the quality of this kind of data has improved markedly, including in Catalonia, where the ICS database is considered a high quality research tool.^4,31,32

Second, external validation analyses testing the performance of the Queralt indices in other populations and settings are needed to better understand its potential generalizability. Importantly, although Queralt’s indices have been developed using data from 8 large public hospitals of Catalonia, they are flexible, i.e., adaptable to other geographical settings, through the identification and weighing of conditions strongly associated with key outcomes (in-hospital death, length of stay, need for ICU stay, healthcare cost, or other outcomes) within a given setting. The software is available online for research purposes, at no cost.³³ Importantly, the Queralt Dx index may perform better in databases such as ICS’s in which pre-existing comorbidities and complications developed during the hospital stay can be differentiated.

Third, in Elixhauser’s index secondary diagnoses related to the principal diagnosis are typically not considered by the score. However, in our analysis we allowed these to be included. This may have overestimated the discriminative ability of Elixhauser’s index, yielding a conservative bias when comparing the performance of the Queralt indices specifically to Elixhauser’s.

Finally, the definition of long hospital stay (as >14 days), although based on the general distribution of duration of hospital stays in our area, may not be completely meaningful in specific subgroups of patients, particularly those with acute, short-duration processes.

Conclusions

Exhaustive comorbidity indices including multiple acute and chronic conditions may have a better performance for risk assessment than often used tools such as Charlson’s and Elixhauser’s. Moreover, our analyses suggest that risk adjustment should go beyond pre-existing comorbidities and include information on principal discharge diagnoses and in-hospital complications. In the era of electronic medical records and of widespread use of automated disease coding systems in hospitalized patients, our findings have important implications for current risk adjustment paradigms, for which exhaustive, comprehensive, automated tools such as the Queralt indices may provide an invaluable, inexpensive opportunity to improve risk adjustment. Validation of these tools in other settings is needed to better understand their generalizability.