Comparing the Performance of Charlson and Elixhauser Comorbidity Indices to Predict In-Hospital Mortality Among a Chinese Population

Introduction

There is a growing trend of and interest in conducting health service and outcome research based on administrative medical databases.¹ Administrative data are routinely collected in hospitals, clinics, pharmacies or other healthcare institutions.² Given that administrative data tend to be large in volume and accessible and provide detailed service utilization information, an increasing number of researchers have been using them to perform cost-effectiveness analysis, risk adjustment, and predict mortality and health outcomes.^3–6

In health outcome research, comorbidities or coexisting medical conditions are one of the most critical factors to adjust.^7,8 The Charlson Comorbidity Index (CCI) and the Elixhauser Comorbidity Index (ECI) are the two best-known indices in the field of patient risk adjustment and outcome prediction.^9,10 The CCI was originally developed in 1987 by Charlson et al,⁹ who reviewed the inpatient hospital charts of 559 medical patients at New York Hospital, along with their 1-year mortality, ultimately defining 17 comorbidities and associated weights, to estimate the mortality risk. In 1998, Elixhauser et al¹⁰ developed a more comprehensive index, of 30 comorbidity measures, by examining a large administrative data of 1,779,167 patients from California. However, the two comorbidity indices were not widely adopted by researchers until 2005, when Quan et al¹¹ proposed the coding algorithms to define and differentiate between the Charlson comorbidities and Elixhauser comorbidities for administrative databases. The study by Quan et al¹¹ allowed researchers to calculate CCIs and ECIs based on administrative databases using either the International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) or the International Classification of Disease and Related Health Problems, 10th Revision (ICD-10). Following the development of CCI and ECI, combined comorbidity indices have also been developed to combine the CCI and ECI. Although a few studies suggest that combined indices have better predictive performance than the CCI and ECI among North American populations, combined indices are relatively new and much less used than the CCI and ECI.^12,13

Comorbidities vary by population characteristics and diseases and have negative effects on patient outcomes, particularly in mortality and readmission percent prediction.¹⁴ Since the original development of these two indices, they have been updated by various researchers and applied to different populations.^15–17 In 2011, Quan et al¹⁵ updated the CCI weights using Canadian data and validated them on data from Australia, Canada, France, Japan, New Zealand and Switzerland. However, their results were based on data in 2008, which may be outdated and not as suggestive as they once were. In addition, they validated only one version of CCI and failed to benchmark the performance of multiple widely used CCIs and ECIs. In 2009 and 2017, van Walraven et al¹⁶ and Moore et al¹⁷ developed two weighting systems for ECI, based on Canadian and American administrative data, respectively. These comorbidity scores have been adopted by various epidemiological studies all over the world.^18–20 Nevertheless, only a few studies have validated or compared the use of these comorbidity indices on non-Caucasian populations, and none of these studies were based on any Chinese populations.^{13–15,21–23}

The large volume of patients in China, together with fast-developing health information systems, furnishes a promising opportunity to enlighten healthcare practice and policy-making based on real-world evidence.^24–27 Although a growing number of researchers are applying the American-population-based CCI and ECI to a Chinese population, there is no evidence of cross-population generalizability of these comorbidity indices using Chinese administrative data.^28–31 Therefore, this study aims to evaluate the predictive accuracy of five commonly used indices in in-hospital mortalities of four disease cohorts based on a large administrative database in Shanxi, China. The five indices include two from the Charlson comorbidity system: the 17 comorbidity indicator variable system by Charlson et al⁹ in 1987 and the single numeric score updated by Quan et al¹⁵ in 2011, and the Elixhauser comorbidity system: the 30 comorbidity indicator variable system by Elixhauser et al¹⁰ in 1998, the single numeric score developed by van Walraven et al¹⁶ in 2009, and the single numeric score developed by Moore et al¹⁷ in 2017.

Materials and Methods

Study Sample

This study used de-identified electronic medical records (EMR) data from the hospital discharge database in Shanxi, China between January 2013 and December 2017. The database included the following information of each patient: demographic characteristics (age, gender and marital status), a primary diagnosis and its ICD-10-CM code, up to 10 secondary diagnoses and their ICD-10-CM codes, up to 7 medical procedures and their ICD-9 procedure codes, and patients’ outcomes (medical costs and discharge outcomes). The ICD-10-CM system in the Chinese EMR database follows international standards, but the last two decimal places may reflect minor changes that have been adapted to encompass local specialized diseases. The study has been approved by the Institutional Review Board (IRB) at Sichuan University, China. Since de-identified data were used in this study, patient consent to review their medical records was not required by the IRB.

Since the predictive abilities of CCI and ECI can vary across different patient outcomes, this study follows the guidelines of Li et al¹⁴ and targets four disease cohorts, including 1) congestive heart failure (CHF), 2) chronic renal failure (CRF), 3) diabetes, and 4) patients who underwent a percutaneous coronary intervention (PCI). CHF patients were identified if their main diagnosis code (ICD-10-CM) contained “I11.0”, “I13.0”, “I13.2”, “I50”. CRF was identified if their main diagnosis codes contained “N18”, and type 2 diabetes patients were identified by “E11.9”. PCI patients were identified if any of their seven procedure codes contained “00.66”, “36.01”, “36.02” or “36.05”. Patients with missing values on gender or age, and those younger than 18 years of age, were excluded from this study. In total, 41,577 CHF patients, 40,419 CRF patients, 171,355 patients with type 2 diabetes, and 39,097 PCI patients were included in this study.

The Outcome and Comorbidities

The primary outcome in this study was in-hospital mortality, defined as all-cause death during the period of hospitalization.¹³ Similar studies have used 30-day mortality, 1-year mortality or unplanned readmission as the outcomes.^9,13,17 However, thirty-day mortality was not available in the Chinese EMR systems at the time of this study. Additionally, readmissions can be identified using the patient’s unique identifier, but the Chinese EMR system, as specified by the former Ministry of Health, did not formulate a way of distinguishing between planned and unplanned readmission.

We used the package “icd” in the statistical environment R to estimate the CCIs and ECIs in this study.³² The package used Quan’s ICD-10 coding algorithms to define Charlson and Elixhauser comorbidity binary variables.¹¹ We did not use the official software package, developed by the Agency for Healthcare Research and Quality, to calculate the Elixhauser index since most of its codes had three decimal places. The Chinese ICD-10 coding system has minor adjustments on the second and third decimal places to local specific diseases. Therefore, this software is not applicable to Chinese electronic medical records. Apart from using the 17 and 30 comorbidity binary variables proposed by Charlson et al⁹ and Elixhauser et al,¹⁰ this study also used three widely used weighting systems that combine comorbidities into single numeric values: 1) Quan’s updated CCI,¹⁵ which ranges from 0 to 24, 2) van Walraven’s ECI,¹⁶ which ranges from −19 to 89, and 3) Moore’s ECI,¹⁷ which ranges from −32 to 99. Higher score on any of these three measures indicates a greater disease burden and a higher probability of having in-hospital mortality.

Statistical Analysis

We calculated the frequencies of the Charlson and Elixhauser comorbidity variables, using ICD-10-CM secondary diagnoses for each disease cohort. Then, we fitted five logistic regression models using Charlson comorbidities, Elixhauser comorbidities and the three numeric scores (listed above) for each patient cohort. The outcome variables in these models are a binary indicator of whether in-hospital death occurred to the patient or not. Each model also included variables such as sex, age, marital status, occupation, length of stay, and hospital levels as covariates. These variables have been conventionally used as covariates to adjust for patient outcomes when a large administrative data set is applied. Biological sex was coded as a binary variable, with the male as the reference group. Age was categorized as 18–45 (reference group), 46–65, 65–75, greater than 75. Marital status was divided into married (reference group), unmarried, widowed, divorced and other. Occupation was categorized as working as a farmer (reference group), jobless, in a private or a public institution, retired and other. Length of stay was categorized into four quartiles with the first quartile as the reference, and hospital level was divided into tertiary and secondary. Tertiary hospitals in China are large national or provincial comprehensive hospitals that provide high-quality specialist services, while secondary hospitals are regional hospitals that provide care for general and less-complicated diseases.²⁸

For each of the four cohorts in this study, we estimated six models with different sets of explanatory variables to predict in-hospital mortality: (1) the baseline model that includes age, gender, marital status, occupation, length of stay, and hospital level (socio-demographic variable sets), and no comorbidity variable is used in this baseline model, (2) baseline model with the Quan comorbidity score,¹⁵ (3) baseline model with the van Walraven comorbidity score,¹⁶ (4) baseline model with the Moore comorbidity score,¹⁷ (5) baseline model with the 17 Charlson comorbidity indicator variables,⁹ (6) baseline model with the 30 Elixhauser comorbidity indicator variables.¹⁰

The concordance statistics, which are often called c-statistics, are a measure of goodness of fit for the logistic regression model. It is equivalent to the area under a receiver operating characteristic (ROC) curve and ranges from 0.5 to 1. The ROC curve illustrates the relationship between the rate of false positives and true positives when selecting the cut-off values for predicting the dichotomous outcomes (in-hospital death in this case). Higher values indicate a better prediction model. We computed c-statistic values to benchmark the predictive performance of these models. 95% confidence intervals (CIs) were computed using 1000 bootstrap replicates for each model. All data cleaning, statistical modeling, and visualization were conducted in the statistical environment R 3.4.1.³³

Results

Table 1 shows the descriptive statistics of patient demographics, covariates, and three numeric comorbidity scores stratified by patient outcomes. Overall, the four patient cohorts had a very low in-hospital mortality rate, with CHF patients having the highest percentage (0.85%) and PCI patients with the lowest percentage (0.03%). CHF patients tended to be older (39.1% older than 75 years of age), had a higher percentage of people who were widowed (6.2%) and farmers (51.4%). By contrast, CRF patients were more likely to be younger (27.4% in the 18–45 age group). PCI patients were much more likely to be hospitalized in tertiary hospitals (97.7%), where most of the high-tech medical equipment was located.

Table 1 Characteristics of the Study Sample Stratified by the Outcome Variable

Table 1 also demonstrates differences in characteristics between dead and alive patients across the four cohorts. Compared with patients who stayed alive during hospitalization, patients who died were older, more likely to be widowed, divorced, retired, and had a lower length of stay in hospitals. The mean values of the van Walraven¹⁶ and Moore¹⁷ score were significantly different between dead and alive patients across the four cohorts. It merits attention that the CCI, updated by Quan et al,¹⁵ showed very little variation across and within the four disease cohorts, indicated by similar means and small standard deviations. Relative to Quan’s CCI,¹⁵ the other two numeric scores (ECI by van Walraven¹⁶ and Moore¹⁷) demonstrates much more within- and across-group variations.

The frequency and percent of the 17 Charlson comorbidities and 30 Elixhauser comorbidities are presented in Table 2. It is worth noting here that some comorbidities were very rare, such as acquired immunodeficiency syndrome (AIDS), alcohol and drug use, consistent with those reported in Li et al.¹⁴ A fair amount of variation in comorbidity proportions can be observed from Table 2 across the four patient cohorts.

Table 2 The Frequency and Percent of Each Comorbidity in Charlson and Elixhauser Comorbidity Systems

The c-statistics of these regression models are shown in Table 3 and Figure 1. The predictive performance of the three numeric comorbidity scores demonstrated different and inconsistent patterns over the four disease cohorts in our sample. The Charlson and Elixhauser comorbidity indicator variable sets, together with the covariates, consistently had higher predictive ability than the baseline and all three numeric score models. Among the three comorbidity scores, the ECI updated by Moore and his colleagues in 2017 outperformed the other two scores (Quan 2011 and van Walraven 2017), except for PCI patients, among which the van Walraven ECI in 2009 outstripped the Moore 2017 ECI. The comorbidity measures, together with sociodemographic variables, generally had the best predictive ability in the diabetes patient cohorts, followed by CRF and PCI cohorts, while they performed the worse among the CHF patient cohort.

Table 3 c-Statistics for Each Model Across Four Disease Cohorts

Figure 1 Comparison of c-statistics among different variable sets.

Discussion

This study compared the performance of five commonly applied comorbidity indices in predicting in-hospital mortality of four disease cohorts based on a homogeneous Chinese population. The results can serve as a guideline for choosing comorbidity indices when performing risk adjustment based on Chinese administrative data in mortality research.

To the best of our knowledge, this is the first study that compares the predictive accuracy of widely recognized comorbidity indices among the Chinese population. Our findings have two major implications. First, using 31 Elixhauser comorbidity indicator variables, instead of relying on combined numeric scores, can generate the largest predictive power, given a sufficiently large sample size. This is consistent with the study performed by Quan et al.¹⁵ Second, when the sample size is relatively small, researchers may be concerned about overfitting and non-convergence problems for Charlson or Elixhauser comorbidity indicator variables since they include 17 and 30 comorbidity dummy variables in the regression model.¹⁶ In that case, our results suggest that the single numeric score by Moore et al¹⁷ in 2017 generally had the highest predictive accuracy, relative to the other two numeric score indices.

Prior to this study, we expected that the Charlson comorbidity score updated by Quan et al¹⁵ in 2011 would demonstrate higher predictive accuracy, relative to the other two numeric scores, as they included a Japanese population and asserted an external validity among an Asian population.¹⁵ However, the Quan 2011 comorbidity score did not perform as well as the other two comorbidity numeric scores, based on our sample. For the diabetes disease and PCI cohorts, the Quan comorbidity score,¹⁵ together with sociodemographic variables, did not even produce a higher c-statistic than those using these sociodemographic variables alone. This indicates that the comorbidity score updated by Quan et al¹⁵ in 2011 may not be generalizable to the Chinese population, despite its inclusion of another Asian population (ie, the Japanese population).

Compared with the single cohort in Moore’s paper,¹⁷ our four cohorts have more variation in patient demographic variables. The CHF cohort includes more elderly patients, while the other three cohorts include more young patients. The prevalence of most comorbidities in this study is comparable to those in van Walraven’s paper,¹⁶ but some comorbidities (AIDS, alcohol abuse, and drug abuse) have a very low frequency. Alcohol and drug abuse is often not considered as a medical symptom or diagnosis in China, which may be attributable to doctors’ coding habits. Underestimating the prevalence of these three comorbidities may result in weaker predicting performance since these variables are indicative of patient outcome. Given the difference in patient characteristics and comorbidity distributions across cohorts, the c-statistics in this study can be better or worse compared to the previous studies.^15–17 For example, our models have higher c-statistics than those in Moore’s study: the c-statistics for the Elixhauser indicator variable model are 0.824, 0.875, 0.923, and 0.879 in our cohorts, while the value is 0.805 in a similar study by Moore et al.¹⁷

EMR data quality is crucial for risk adjustment, patient outcome prediction, and hospital performance profiling. Prior validation studies of comorbidity scores, using administrative databases, examined the accuracy of disease coding. In the electronic medical record system in Shanxi, China, doctors fill out the name of diagnosis, surgery, and their corresponding codes. The department of medical records conducted data quality control and verified a 5% random sample of all patient records every month. Those medical records with incomplete information, miscoding, or logical errors were returned to the physicians and these physicians were subsequently retrained. Despite the aforementioned measures to improve EMR data quality, there have been no formal studies assessing the accuracy of data as Hsia et al did.^34,35 Hsia et al suggested a potential concern related to overcoding for reimbursement claim but it does not apply to administrative data from China because the Chinese health care is still a fee-for-service-based system.

The other issue about data quality is the number of secondary diagnoses in this Shanxi database. The administrative database in other countries normally has over 15 secondary diagnoses,¹⁵ and the National Inpatient Sample (NIS) provided by the Healthcare Cost and Utilization Project (HCUP) in the United States offers over 30 diagnoses.³⁶ By contrast, the Shanxi EMR database only provides one primary diagnosis and up to 10 secondary diagnoses, based on a formulation developed by the former Ministry of Health in China.³⁷ This could lead to an underestimation of comorbidities. Some patients in this study can have more than 10 comorbidities, but those comorbidities may not all be recorded. Therefore, this may have resulted in residual confounding in our study findings. The low frequency of some comorbidities, such as acquired immunodeficiency syndrome (AIDS), alcohol and drugs, also merits discussion. Despite its low prevalence of HIV/AIDS in our study sample, the HIV/AIDS diagnosis was accompanied by a routine blood test upon admission to the hospital. For drug and alcohol abuse, we would expect extremely low frequency in our sample since these two comorbidities are rarer in the Chinese population, relative to the North American population.³⁸ As reported by the World Health Organization, the average yearly alcohol consumption in China was 6.7 L/person (2.2 L for women, 10.9 L for men), compared to 9.2 L/person (4.9 L for women, 13.6 L for men) in the United States.³⁸ Nonetheless, these low-frequency comorbidities corroborated with findings from Li et al.¹⁴

This study should be interpreted with caution because there are several limitations. First, we had no follow-up data on patients’ outcomes. The mortality percent in our study may be underestimated because terminally ill Chinese patients with little hope of recovery tend to withdraw from hospitals, either for reasons related to cultural habits or financial affordability.^39,40 Another limitation is the representativeness or external validity of our sample. Since there is no publicly available nationwide inpatient sample in China like the NIS, researchers interested in Chinese EMR must rely on their collaboration and connection with a given local health administration agency to gain access to the data. Although the National Health and Family Planning Commission in China holds a Nationwide Hospital Discharge Database, which routinely collects EMR data from most secondary and tertiary hospitals,⁴¹ its use and access are highly restricted to a limited number of researchers. Third, since this is an observational study that relies on secondary data, it is subject to potential information bias and residual confounding caused by inaccurate coding, unobserved patient or hospital characteristics.

Conclusions

The large population and rapid development of an electronic medical record system provide a unique opportunity to inform healthcare practice and policymaking. Our study suggests that Elixhauser comorbidity indicator variables should be used when a large Chinese electronic medical record database is available since they have the best predictive performance among the five indices, while Moore’s score system is more appropriate when only a relatively small database can be accessed. Besides, Moore’s score system can be more intuitive and informative in characterizing the comorbidity distribution of the population. Results from this work serve as a reference for researchers in China when they select comorbidity measures for health outcome prediction and risk adjustment. Besides, the results may also be informative to researchers who are interested in those electronic medical record databases with a high number of Chinese patients.