Validation and Comparison of a Modified Elixhauser Index for Predicting In-Hospital Mortality in Italian Internal Medicine Wards

Introduction

Prevalence of people living with multimorbidity is increasing,¹ especially in high-income countries during the last years.² Multimorbidity is defined as the detection of two or more chronic conditions in a single individual,³ and individuals with combinations of different conditions require a comprehensive management approach.⁴ Although multimorbidity increases risk of death^5,6 and both number and different combinations of conditions have been reported to rise mortality,^6–8 its definition is still a matter of debate.^9,10 Despite these evidences, health-care systems are organized for managing single diseases rather than their combinations because of their focus on specialization.¹¹ In the last decade, our research group published a series of papers using administrative databases in order to demonstrate the relationship between in-hospital mortality (IHM) and comorbidity in different conditions such as myocardial infarction, pulmonary embolism, stroke, urinary tract infections, renal transplantation, chronic obstructive pulmonary disease and acute esophageal variceal bleeding.^12–19 Subsequently, we decided to modify an existing score in order to adapt it to internal medicine patients, and, in 2017, we obtained a new index of comorbidity based on internal medicine International Classification of Diseases codes and derived from Elixhauser index,²⁰ to predict risk for IHM in patients admitted to internal medicine wards.²¹

In order to obtain information on comorbidity health care professionals could use medical chart reviews or administrative data. The first method is time consuming and laborious, whilst analysis of administrative databases is fast and cheap. On the other hand, the number of comorbidities identified with administrative databases is lower than those detected by chart review. Aim of this study was to validate this score comparing results derived from chart review and administrative data in a consecutive cohort of individuals admitted to internal medicine. Moreover, a prognostic index calculated from clinical data previously tested in internal medicine patients was used as a comparator score.

Patients and Methods

Population and Administrative Data Source

This retrospective study was conducted in agreement with the declaration of Helsinki stated in 1975, and revised in 2013, and it was approved by the local institutional committees for human research (Comitato Etico Azienda Ospedaliera Universitaria di Ferrara, approvazione 100,893).

The study included all hospital admissions for different causes along a 6-month period from January to June 2016, recorded in the database of the University Hospital St. Anna of Ferrara, Region Emilia-Romagna (RER) of Italy, and maintained by the Centre for Health Statistics. Since 1999, our hospital started to use an electronic database to store all the discharge hospital records (DHR) of hospitalized patient. These DHR report, gender, date of birth, date and department of hospital admission and discharge, vital status at discharge, length of stay, charge details, main and up to 6 accessory discharge diagnoses, and the most important diagnostic procedures, based on the International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM). Moreover, authors evaluated all medical charts and recorded clinical data and comorbidity after patients’ discharge. In agreement with national dispositions by law in terms of privacy, the health authorities removed patient names, exact addresses, and other potential identifiers from the database provided for this study.

The St. Anna University Hospital is a 660-bed teaching hospital with all facilities, excluding only cardiothoracic surgery. The majority of admissions are from the province of Ferrara, they are approximately 27,000 yearly, and the hospital is the hub center of the area. The province of Ferrara (≈ 350,000 inhabitants, mean age 47.3 years, 25.9% >65 years) is served by one Teaching Hospital (hub center) and three community hospitals (spoke centers about 200 beds/each). Local economy is mainly agriculture-based, and in minor part industrial. The annual flow of patients by the emergency department (ED) is approximately 90,000, mainly elderly subjects due to the fact that the area is characterized by a high percentage of elderly subjects, and approximately 3000 subjects are aged more than 90 years. The Department of Medicine consists of four Internal Medicine units, two Infectious Disease units, and one each of Geriatrics, and Gastroenterology (165 total beds, 24/24 h and 7/7 days open to the ED admissions). About one-third of all hospital admissions are directed to the Department of Medicine. The great part of medical and nursing staff is permanent, covering also festive days or holidays.

Comorbidity Score Calculation

We calculated a modified Elixhauser index, a novel score recently proposed by our group in order to take into due account the comorbidity burden.²¹ For development of the score, we included hospital admissions for any cause in the department of internal medicine. At the time of the score development, more than 75,000 admissions aged 72 years were analyzed, and administrative data were used following guidelines suggested by Quan et al.²² The points assigned to each condition selected in the score ranged from 0 to 16, and the possible range of the score varied between 0 and 89. The following conditions were considered for score calculation: age (0–60 years points=0; 61–70 years points=3; 71–80 years points=7; 81–90 years points 11; ≥91 years points 16), gender (male points=2), presence of renal diseases (points=1), neurological disorders (points=3), lymphoma (points=4), solid tumor with metastasis (points=4), ischemic heart disease (points=5), congestive heart disease (points=5), coagulopathy (points=8), fluid and electrolyte disorders (points=8), liver disease (points=10), weight loss (points=11), and metastatic cancer (points=12).²¹ The risk of IHM became significant when the score was >40, overcoming the value of 60%. In this comparison study, the modified Elixhauser score was calculated using both administrative data recorded at the time of discharge and electronic medical charts. Authors evaluated all medical charts and recorded comorbidity according to the modified Elixhauser score and according to a previous prognostic index derived from clinical data proposed by Cei et al in 2015 in elderly medical patients.²³ For the calculation of this latter index male gender (point=1), dependency in activities of daily living (1–4 ADLs points=2; 5 ADLs points=5), congestive heart failure (points=2), solitary or hematologic cancer (points=3), metastatic cancer (points=8), serum creatinine >3 mg/dl (points=2), serum album <3 g/dl (points=2) or between 3 and 3.4 g/dl (point=1) were evaluated.²³

Statistical Analysis

Analysis was limited to hard clinical indicators: fatal (IHM) and nonfatal (patient discharged alive) outcome. We performed a descriptive analysis of all data collected, and results were expressed as absolute numbers, percentages, and mean ± SD. Univariate analysis was performed to define the difference between survivors and deceased subjects; a statistical analysis was performed using the Chi-square test, Student t test, and Mann–Whitney test as appropriate. We compared models using receiver operating characteristics (ROC) curve scores (c-statistics). The probability of risk of death assigning is the c-statistics, if the value is 0.5 the model is not a good predictor, while a value of 1 suggests a perfect discrimination between deceased and survivor individuals. Values lower than 0.7 are considered poor predictors and those between 0.7 and 0.8 could be reasonable. When the value is greater than 0.8 prediction is good. The area under the receiver-operator characteristic curve (AUC) and 95% confidence intervals were reported.

SPSS 13.0 for Windows (SPSS Inc., Chicago, IL, 2004) was used for statistical analyses.

Results

During the study period, 1962 individuals were admitted in the medical department (average 10.8 patients per day) of whom 348 were excluded because data were incomplete. Totally, 1614 subjects were analyzed, 43.3% were male and mean age was 79.1±13.1 years. IHM was recorded in 224 patients (13.9%). Main characteristics of the population are reported in Table 1, while prevalence of the different conditions considered in order to calculate the different scores using both administrative data recorded at the time of discharge and electronic medical charts is reported in Table 2. The number of comorbidities derived from electronic medical charts was higher than comorbidities derived from administrative data recorded at the time of discharge (3787 vs 1875), and modified Elixhauser score calculated from electronic medical charts had an average value of 5 points higher than modified Elixhauser score calculated from administrative data recorded at the time of discharge. Age and comorbidity scores were higher in deceased than in survivors (Tables 3 and 4). The modified Elixhauser score calculated by administrative data and by chart review and the comparator one was 18.13±9.36 vs 24.43±11.27 vs 7.63±3.3, respectively, and the c-statistic was 0.758 (95% CI 0.727–0.790), 0.811 (95% CI 0.782–0.840) and 0.740 (95% CI 0.709–0.771), respectively. AUC of the different scores are reported in Figure 1. Values of modified Elixhauser score calculated by administrative data between 44 and 53 suggested a risk of IHM between 50% and 75%, whilst when the calculated value was higher than 53 risk of IHM was higher than 75% (Figure 2).

Table 1 Main Characteristics of the 1,614 Individuals Investigated

Table 2 Comparison of Prevalence of the Different Conditions Considered in Order to Calculate the Different Scores Using Both Electronic Medical Charts and Administrative Data Recorded at the Time of Discharge

Table 3 Main Characteristics of Deceased and Survivors

Table 4 Comparison of Prevalence of the Different Conditions Considered in Order to Calculate the Different Scores Using Electronic Medical Charts and Administrative Data Recorded at the Time of Discharge in Deceased and Survivors

Figure 1 Receiver operating characteristics (ROC) analysis reporting the area under the receiver-operator characteristic curve of the different scores evaluated.

Figure 2 Relationship between values of modified Elixhauser score calculated by administrative data and risk of in-hospital mortality (IHM).

Discussion

In this paper, we aimed to validate a modified Elixhauser score developed in 2017 by our group²¹ and we found that its performance could be considered reasonable if calculated using administrative data recorded at the time of discharge and good if calculated using electronic medical charts. Moreover, using electronic medical charts its performance was better than a prognostic index derived from clinical data and evaluated by Cei et al in 2015 in a similar setting;²³ the reason for comparing the modified Elixhauser score with the one evaluated by Cei et al is due to the setting, ie, Italian internal medicine wards, and the good performance reported by Cei et al that was 0.81 (95% CI 0.78–0.84).²³ We found that our modified Elixhauser score performed similarly to Cei et al²³ clinical prognostic index, and when it was calculated using clinical data derived by charts review its prognostic performance improved due to the higher number of comorbid conditions detected reviewing every single case.

We previously evaluated the usefulness of the modified Elixhauser score on risk factors for IHM in internal medicine wards patients with infectious diseases. We included all hospital admissions recorded in the database of the local hospital and ICD-9-CM codes were selected to identify infections, development of sepsis, and to calculate the score. The modified Elixhauser score was independently associated with IHM and the receiver operating characteristic (ROC) analysis showed that the area under the curve was 0.724 for detection of IHM.²⁴

In elderly population, acute medical condition requiring hospitalizations contributes to decline of health status,²⁵ and adjustment for comorbidities could be useful for risk prediction of negative outcomes. The use of comorbidities in risk adjustment for health outcomes research is frequently necessary to explain some of the observed variations. Medical chart reviews to obtain information on comorbidities is time consuming and laborious. Administrative databases have provided an alternative for health services researchers to obtain comorbidity information, however, the rates obtained from databases are still a matter of debate.

In 2003, de Groot et al²⁶ systematically reviewed available methods to measure comorbidity and to assess their validity and reliability. Content, concurrent, predictive and construct validity, and the reliability were assessed. The authors found that Charlson Comorbidity Index (CCI) was the most extensively studied score for predicting mortality, while Cumulative Illness Rating Scale (CIRS) addressed all relevant body systems without using specific diagnoses, Index of Coexisting Disease (ICED) measured disease severity and disability, Kaplan Index was specifically developed for use in diabetes research. All were considered valid and reliable methods to be used in clinical research.²⁶ Subsequently in 2012, it was carried out a systematic review in order to identify the best performing comorbidity indices for short-term and long-term mortality.²⁷ Authors analyzed 54 articles and found that the Deyo variant of CCI was the most commonly referred comparator followed by the Elixhauser measure. Deyo variant of CCI represents the CCI adaptation for ICD-9-CM.²⁸ Comorbidity predicted in a better way long-term than short-term mortality and Elixhauser seemed to be the best predictor for this outcome. In the case of short-term mortality, recalibration giving empirical weights was reported to be more important than the choice of comorbidity measure.²⁷ In fact, our modified Elixhauser score was developed weighting differently conditions impacting survival of patients during admission.²¹

As expected, chart review data turns out to perform better in predicting outcome of patients than administrative information. Similar performance was showed in 1992 by Hannan et al²⁹ that compared the ability of a clinical and administrative database in New York State to predict IHM and to assess hospital performance for coronary artery bypass graft surgery. They found that clinical data were better than administrative ones in predicting mortality.²⁹ Moreover, in 1999 Kieszak et al³⁰ compared CCI calculated from medical record data and ICD-9-CM in order to determine how well inpatient and 30-day mortality, length of stay, and complications were predicted among Medicare beneficiaries hospitalized for carotid endarterectomy. Using logistic regression, CCI derived from clinical chart review was found to be a significant predictor for all clinical outcomes, after controlling for age, gender, and neurologic and medical risk factors. In 2006 Martins et al³¹ compared the predictive capacity of the original CCI, the CCI with new assigned diagnostic codes and estimated weights, and a new developed comorbidity index in a Brazilian population; the c-statistic was 0.72 for the original CCI, and increased to 0.74 for the CCI with new weights and 0.76 for the new index. The c-statistic increases in all the comorbidity indices with the utilization of more diagnostic information.³¹ In 2007 Luthi et al³² compared cross-sectionally the CCI derived from a rapid single-day chart review with the same index derived from administrative data (ICD-10-CM) to determine how well each predicted IHM and nosocomial infection. Authors evaluated 890 adult patients hospitalized from acute care wards in Switzerland and found that CCI derived from administrative data provided a higher c-statistic compared with single-day chart review for IHM (0.863 vs 0.795) and for nosocomial infection (0.645 vs 0.614). They concluded that CCI derived from administrative data was superior to the index derived from rapid single-day chart review.³²

In administrative database analysis, an important issue that should be taken into account is the lower number of comorbidities identified compared to those recognized by charts review. In our study, we detected 1875 comorbidities using administrative records and 3787 using charts review. In 2011, Chong et al³³ quantified retrospectively and cross-sectionally the agreement between administrative data and medical charts review in determining the presence of comorbidities. Moreover, authors examined which factors were associated with under- or over-reporting of comorbidities using administrative data. They analyzed patients aged ≥55 years, hospitalized for pneumonia at 3 acute care hospitals. They reported that prevalence of almost all comorbidities obtained using administrative data was lower than that obtained using medical charts review. Agreement between comorbidities obtained from medical charts and administrative data varied widely. Factors associated with over-reporting of comorbidities were increased length of hospital stay, disease severity, and IHM. On the contrary, those associated with under-reporting were number of comorbidities, age, and hospital admission in the previous 3 months. They concluded that the validity of using secondary diagnoses detected from administrative data as an alternative to medical charts for identification of comorbidities is related to age, number of comorbidities, hospital admission in the previous 3 months, severity of illness, length of hospitalization, and IHM.³³ In the same year, Christensen et al³⁴ examined the performance of comorbidity scores in predicting mortality of intensive care unit (ICU) patients. They evaluated CCI alone and in combination with other readily available administrative data and three physiology-based scores (acute physiology and chronic health evaluations [APACHE] II, simplified acute physiology score [SAPS] II, and SAPS III) in predicting short- and long-term mortality following intensive care. Data for CCI calculation, age and gender, surgical/medical status, social factors, mechanical ventilation and renal replacement therapy, primary diagnosis, and complete follow-up for 1-year mortality were obtained from administrative databases. When CCI was combined with other administrative data c-statistics was 0.75 for IHM, 0.75 for 30-day mortality, and 0.72 for 1-year mortality. Difference between c-statistics of physiology-based systems and the CCI combined with other administrative data was not significant. Authors concluded that CCI combined with administrative data could predict short- and long-term mortality for ICU patients in a similar way as physiology-based scores.³⁴ Quan et al³⁵ re-evaluated the CCI and reassigned weights to each condition by identifying and tested predictivity of mortality within 1 year after hospital discharge. Authors applied the updated index and weights to administrative databases from 6 different countries and tested for their ability to predict IHM. C-statistics for detecting IHM mortality between the new score and the original CCI were 0.825 and 0.808, respectively, in Australia, 0.828 and 0.825 in Canada, 0.878 and 0.882 in France, 0.727 and 0.723 in Japan, 0.831 and 0.836 in New Zealand, and 0.869 and 0.876 in Switzerland.³⁵

Limitations

Several limitations should be mentioned. First of all, this is a single-center retrospective study, therefore results could not be generalizable. Secondly, we did not differentiate between conditions existing before admission and complications developed during hospitalization; moreover, we did not take into consideration admission diagnosis. Third, the modified Elixhauser index did not include all cardiovascular diseases, nor a very common illness such as chronic obstructive pulmonary disease, that could be frequently encountered in patients admitted to internal medicine wards.^36,37 Outcome of these conditions is highly related to comorbidity.^36,37 Fourth, our outcome was IHM, we had no information about out-of-hospital mortality. Originally, the score used in this study was developed from administrative data that are claimed for financial reasons and not for research, therefore data quality could be different in different health care organization and hospital setting.³⁸ Performance of clinical physicians at the time of coding and financial incentives such as diagnosis-related groups or healthcare-related groups could be different between hospitals depending on gaps in clinical information and the billing pressure could alter the quality rating of administrative data, as underlined by our comparison with data obtained from chart review. We believe that errors of omission were the leading cause of different performances of modified Elixhauser score calculated by administrative data and by chart review.³⁹ Interrater reliability could not be assessed because only one author reviewed medical chart, besides we compared scores with different number of items (13 vs 9). On the other hand, both scores had been tested in similar Italian hospital settings.

Conclusion

Physicians mainly focus their attention on diagnosis and treatment, however nowadays, due to the aging of the population, prognosis is becoming a major responsibility, therefore, risk stratification has to be taken into account in everyday clinical practice. Comorbidity is often considered as a complication in specialized clinical conditions, however, it could be that associations of different chronic illnesses determine the outcome of different treatment.^40–42 Diagnostic and therapeutic strategies should be influenced by prognosis, especially in the case of shortage of resources.

Our study allowed validation and comparison of a new prognostic score derived from Elixhauser index. Calculation was based on both chart review and administrative data in order to identify the quoad vitam prognosis of patients admitted to internal medicine wards. The prognostic power of such a score increased when it was calculated by chart review instead of using administrative data due to the higher number of comorbidity that could be identified. However, when it was calculated by using administrative data its prognostic efficiency was similar to a previous index calculated in a similar clinical setting by direct assessment of the patient’s comorbidities.