Prediction Ability of Charlson, Elixhauser, and Rx-Risk Comorbidity Indices for Mortality in Patients with Hip Fracture. A Danish Population-Based Cohort Study from 2014 – 2018

Introduction

Hip fracture is a common condition among elderly.¹ Patients are at high risk of developing postoperative complications, increased disability, and have a 1-year mortality of approximately 30%.^1–3 Furthermore, comorbidities are common among hip fracture patients, with more than 75% of patients having a chronic disease. Patients with higher comorbidity burden are at higher risk of mortality and postoperative complications.^3,4 In observational studies, comorbidity indices are widely used for confounder adjustment among hip fracture patients. Two frequently used diagnosis-based comorbidity indices are the Charlson comorbidity index^5–8 (CCI) and the Elixhauser comorbidity index (ECI).^7–10 Diagnosis-based comorbidity indices have the disadvantage of being subject to variations in coding of diagnoses both between clinicians and between Hospitals. As well as, a lack of diagnoses from outpatient clinics and general practices.¹¹ Therefore, prescription-based comorbidity indices has been developed¹² such as the Rx-risk index. Rx-risk index could capture both severe but also less severe conditions not requiring hospital contact and diagnosis.

One previous study has investigated the prediction ability in terms of discrimination of CCI and ECI for mortality in hip fracture patients. However, this study only included patients operated with arthroplasty⁸ for a femoral neck fracture, not including patients with a sub- or per-trochanteric fracture, which account for about 50% of all hip fracture patients.¹³ Nevertheless, the study concluded that a model with age and gender alone had superior discrimination of mortality compared with complex CCI and ECI.⁸ The Rx-risk index has not previously been investigated in discriminating mortality among hip fracture patients. Thus, it remains unclear which comorbidity index is most accurate in predicting the mortality of hip fracture patients. Therefore, it is unknown which comorbidity index is best for confounder adjustment in epidemiological observational studies based on registry data. Better control for comorbidity confounding might lead to increase in internal validity of future epidemiological studies of hip fracture patients and improve basis for clinical decision making. We hypothesized that more complex indices may improve on the capture of patient’s preoperative comorbidity and prediction of mortality compared to a simple measure based on a few variables such as age and gender.

The aim of this study was to compare predictive ability of CCI, ECI and Rx-risk index individually and in combination for 30 day- and 1 year mortality in a hip fracture surgery population. We focused on two traditional measures of the quality of prediction, thus discrimination and calibration of indices.¹⁴

Method

Study Design, Setting and Participants

Our cohort study was conducted using prospectively collected data from Danish medical registries, encompassing the entire Danish population, which has a tax-supported health care system with free access to care.¹⁵ All patients ≥ 65 years admitted to a hospital with a first-time femoral neck (S720), pertrochanteric, (S721), or subtrochanteric (S722) femoral fracture treated surgically with osteosynthesis or total/hemi arthroplasty from 2014 to 2018 were included from the population based clinical quality database; the Danish Multidisciplinary Hip Fracture Registry (DMHFR).¹³

Comorbidity Measures and Data Sources

CCI consists of a weighted score based on 19 conditions and was developed using information from 559 breast cancer patients.⁵ The ECI is a comorbidity measure developed with the purpose to use in large administrative inpatient dataset. The ECI includes a comprehensive list of 21 comorbid conditions and is summarized by a weighted score for each comorbid condition.⁹ In the current study, the CCI and ECI were defined using International Classification of Diseases ICD-10 diagnosis codes from the Danish National Patient Registry (DNPR) covering all Danish hospitals.¹⁶ Primary and secondary discharge diagnoses from in-hospital and outpatient clinics with a 10-year lookback period, and with a discharge prior to hip fracture admission were included. Diagnoses recorded during the hip fracture admission were excluded due to the possibility of being complications to hip fracture treatment.

The Rx-risk index was initially developed to measure healthcare expenses, and contains 29 categories of comorbidity based on Anatomical Therapeutic Chemical Classification System codes (ATC), summarized by a weighted score for each comorbid condition.¹⁷ In the current study, the Rx-risk index was defined by ATC codes from The Danish National Prescription Registry with a one year lookback period.¹⁸

The Danish Civil Registration System (DCRS) assigns all Danish residents with a unique identification number. It is recorded at all contacts with the healthcare system and is included in all Danish registries. Therefore, an unambiguous linkage between all medical registries is possible¹⁵.

A full list of diagnosis and ATC codes is found in Supplementary Tables 1–3.

Outcome

The outcome was death within 30 days and 1 year. Follow-up started on the day of surgery.

Statistical methods

The following comorbidity indices prediction ability were compared to age and gender model (base model): CCI, ECI and Rx-risk index. All comorbidity indices were investigated both as continuous variables and in categories in terms of predictive ability for mortality using logistic regression, assuming each observation was independent. The additional predictive ability of pairwise combinations of two distinct indices was investigated, such as the combination of CCI and Rx-risk index, the ECI and Rx-risk index, and CCI and ECI. The base model was used as reference.

For each model we assessed two traditional measures of the quality of prediction, thus, discrimination and calibration. The discrimination answers the question of how well the model can separate the persons with and without outcome (death). The discrimination should be interpreted as the probability that a randomly selected patient who experience the outcome (death) had a higher comorbidity score than a person who had not experience the outcome. Each model’s discrimination ability was illustrated graphically with receiver operating characteristic (ROC)-curves.¹⁹ To construct an ROC curve, the sensitivity (true positive rate) was plotted against the 1-specificity (false positive rate) for consecutive cutoff points for the probability of death. If the model has perfect discrimination, the ROC curve should reach the upper left corner of the plot (corresponding to 100% sensitivity and 100% specificity). The area under the ROC curves is equivalent to c-statistic which we calculated with the logistic regression model. In addition, the contrasts in c-statistic to the base model as well as the respective 95% confidence intervals (CI) were calculated. The c-statistic ranges from 0 to 1, with 1 indicating perfect discrimination and 0.5 reflecting chance.²⁰

We assessed calibration of comorbidity models with 10-percentiles graphically, which answer the question of how well the predicted probability of death agrees with observed mortality.¹⁴

The association between the individual components included in CCI, ECI and Rx-Risk and mortality was evaluated by calculating odds ratios (OR) adjusted for age, gender and the remaining index components. Thereby, comparing patients with and without specific individual comorbidity component.

The prediction models were not tested using training and test-data, given that they were not newly developed, but already well-established prediction models.

The content of this paper follows the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines and the REporting of studies Conducted using Observational Routinely-collected Data (RECORD) guidelines.²¹ Analyses were performed using SAS V. 9.4 (SAS Institute Inc., Cary, NC, USA) and R version 3.6.1 (R Foundation for Statistical Computing).

The study was reported to the Danish Data Protection Agency through registration at Aarhus University (record number: AU-2016-051-000001, sequential number 880). Non-interventional registry-based research projects that do not involve human biological material but are based on pure data such as numbers do not require notification to the Danish Scientific Ethics Committee.²²

Results

The study cohort consisted of 31,443 patients of whom the majority were female (69%), and the most frequent age category were older than 85 years (42%) (Table 1). A total of 10.1% (n=3177) patients died within 30 days, while 26.6% (n=8366) patients died within 1 year. Further description of the study population and distributions of comorbidity measures in categories are provided in Table 1.

Table 1 Characteristics of 31,443 Patients Undergoing Hip Fracture Surgery from 2014 to 2018

30-Day Mortality

The base model had a discrimination ability of 30-day mortality with c-statistic=0.70, CI: 0.69 −0.71. Adding CCI, ECI, Rx-Risk individually to the base model resulted in higher c-statistics of 0.72 (CI: 0.71–0.73), 0.72 (CI: 0.71–0.73), and 0.73 (CI: 0.72–0.74), respectively. A combination of the CCI and the Rx-risk index had the best 30-day mortality discrimination ability (c-statistic=0.74, CI: 0.73–0.75). Contrasts in c-statistics compared to the base model are provided in Table 2. Figure 1 illustrates the ROC curves for 30-day mortality for CCI, ECI, and Rx-Risk as continuous scores.

Table 2 C- Statistic and C-Statistic Contrast for Comorbidity Indicies Predicting 30-Day and 1-Year Mortality Among 31,443 Hip Fracture Patients

Figure 1 Receiver operating characteristic (ROC) curves for 30-day mortality.

All comorbidity indices had similar discrimination ability if included as a continuous score or as in categories (Table 2). In addition, calibration of base model and all individual comorbidity indices was similarly good (Supplementary Figure 1).

1-Year Mortality

The base model had a discrimination ability of 1-year mortality with c-statistic=0.68, CI: 0.67 −0.69. Adding CCI, ECI, Rx-Risk individually to the base model resulted in higher c-statistic of 0.71 (CI: 0.71–0.72), 0.71 (CI: 0.70–0.72), and 0.71 (CI: 0.71–0.72), respectively. A combination of the CCI and the Rx-risk index had the best 1-year mortality discrimination ability (c-statistic=0.73, CI: 0.73–0.740). Figure 2 illustrates the ROC curves for 1-year mortality for CCI, ECI, and Rx-Risk as continuous scores.

Figure 2 Receiver operating characteristic (ROC) curves for 1-year mortality.

All comorbidity indices had similar discrimination ability if included as a continuous score and in categories (Table 2). In addition, calibration of base model and all individual comorbidity indices was similarly good (Supplementary Figure 2).

Components of the Comorbidity Indices

When looking at patients with the different components of CCI, ECI and Rx-risk index, patients having AIDS/HIV had the highest 30-day relative mortality with an adjusted OR of 3.59 (CI: 0.95–13.61) in CCI. While patients with metastatic tumor had the highest 1-year relative mortality with an adjusted OR of 4.33 (CI: 3.57–5.25). The full list of OR of the single components are found in Tables 3 and 4.

Table 3 Odds Ratio (OR) for 30-Day Mortality Among 31,443 Hip Fracture Patients. Comparing Patients with and without Specific Components of Charlson Comorbidity Index, Elixhauser Comorbidity Index and Rx-Risk Index. Adjusted for Age, Gender, and the Remaining Index Components of the Index of Interest

Table 4 Odds Ratio (OR) for 1-Year Mortality Among 31,443 Hip Fracture Patients. Comparing Patients with and without Specific Components of Charlson Comorbidity Index, Elixhauser Comorbidity Index and Rx-Risk Index. Adjusted for Age, Gender, and the Remaining Index Components of the Index of Interest

Discussion

This study evaluated the predictive ability for mortality, including both the discrimination and calibration of different comorbidity indices among hip fracture patients in Denmark. Good discrimination was achieved by using a base model (age and gender) (c-statistic was 0.70 and 0.68 for 30-day and 1-year mortality, respectively). The discrimination was improved by adding comorbidity-indices individually or in combination to the base model. The best discrimination was for CCI combined with the Rx-risk index and base model (c-statistic was 0.74 and 0.73, for 30-day and 1-year mortality, respectively). However, the improvement was marginal. Calibration of indices was similar.

Methodological Considerations

In our study, we included both in-hospital and outpatient diagnoses reducing the potential of underreporting of specific diagnosis codes and ensuring a completeness of data. The Rx-risk index is based on pharmacy dispensings, which have good registration validation and completeness.¹⁸ However, prescribed but failed to fill-in medication was not included, which has shown to account for about 10% of general practitioners prescriptions in Denmark.²³ As well as patient compliance after the dispensing is unknown. Data were collected by a variety of clinicians as part of daily routine work. Therefore, coding errors of specific diagnosis codes are unavoidable. However, coding errors will be distributed evenly in large registers, which will lead to non-differentiated misclassification. Lastly, our study was performed on a hip fracture population in Denmark, which might be different from other populations.²⁴ Generalizing our results on other patient groups or geographical areas, should be done with caution.

Interpretation

CCI, ECI and Rx-risk index have been compared in different settings.^{7,8,11,25–28} Some studies have compared indices different from ours.^7,11,27 Other studies have evaluated indices in cohorts not entirely comparable to ours^25–28 such as cohort of arthroplasty patients. Hip fracture patients are older, more frail and comorbid, acutely operated compared with elective arthroplasty patients, and mortality risk is substantially different in hip fracture versus arthroplasty patients. The most comparable study to ours is a Swedish study by Bülow et. al.⁸ They reported c-statistics for 30-day mortality of 0.67 (95% CI: 0.67–0.68) for ECI and 0.69 (95% CI: 0.68–0.69) for CCI, both including age and gender. These c-statistics were slightly lower than in the present study, which may be due to differences in patient characteristics, since Bülow et al included all age groups and only patients treated with arthroplasty due to hip fracture. Furthermore, Bülow et al only used diagnoses 1-year lookback period to form the comorbidity indices, while we used diagnoses 10 years prior. We used a 10-year lookback period, since this has been shown to discriminate 1-year mortality slightly better.²⁹

Only few previous studies find a strong discrimination ability for mortality with a c-statistic >0.8. This may be due to that the index has either been developed for a specific context other than the investigated or developed as a “one-fits-all” index. Although, a perfect comorbidity index to predict mortality is difficult to obtain since other complex factors affects mortality,^30,31 it is important to identify the index with the best c-statistic in order to control for confounding and decrease bias due to comorbidity in epidemiological studies as much as possible.³²

Our study findings regarding the single components of the different indices may improve our knowledge towards generating a new comorbidity index with a strong prediction ability for mortality among hip fracture patients. First, we found a series of components which had a protective effect on mortality, such as migraine, rheumatic disease and para-/hemiplegia. This may be contributed to surveillance bias, since patients with these diseases will be followed closer by health professionals and therefore have a lower threshold for identifying and treating other milder cases of comorbidities. Secondly, we found that the weighting of some components was in discrepancy with the OR for mortality when seeing the single component as an exposure for mortality. For example, dementia in CCI is weighted as 1, even though dementia has an OR more a like the components weighted with 2 or 6. The discrepancy in the weighting and the OR for mortality may happen due to more effective treatment available today, which would lower mortality compared to 1987 when the CCI was developed. Lastly, we found a series of components which increased the OR for death. Most of which is well known to increase mortality in a geriatric hip fracture population, such as cancer,^33–35 cardiovascular disease,^33–35 dementia,^33–35 chronic airways disease^33–35 as well as liver- and renal diseases.^33–35

To summarize, we found the highest 30-day- and 1-year mortality discrimination ability when combining CCI and Rx-risk with age and gender (c-statistic=0.74 and c-statistic=0.73, respectively). However, age and gender alone had almost the same discrimination ability (c-statistic=0.70, CI: 0.69–0.71 and c-statistic=0.68, CI: 0.67 −0.69, respectively) as complex combined indices. Our findings indicate that 1) it will be possible to develop a new comorbidity index, combining data on both diagnosis and pharmacy dispensing specifically for hip fracture patients, 2) the need for investigating weighting of the components in a hip fracture population, as they might be different from original comorbidity indices weightings. At last, hip fracture patients are often elderly and frail and adjusting for complex comorbidity index will not remove all confounding since other factors are associated with mortality. The inclusion of other domains such as level of function (i.e activities of daily living), cognition, social support, and nutrition would likely further improve prediction ability of comorbidity indices and controlling for confounding.^30,31