Comparative Performance of Comorbidity Measures in Predicting Health Outcomes in Patients with Chronic Obstructive Pulmonary Disease

Introduction

Chronic obstructive pulmonary disease (COPD) has tremendous health impacts worldwide.^1–3 It is symbolized by chronic respiratory symptoms and persistent airflow limitation.⁴ Because of the link between COPD and systemic inflammation, several comorbidities, such as cardiovascular and cerebrovascular diseases, usually coexist with COPD and are significantly associated with poor prognoses.^4–8 Research has indicated that ischemic heart disease, heart failure, stroke, and the number of comorbidities are associated with a 1.2 to 3.4-fold increased risk of mortality.^5–8 Therefore, if we could measure comorbidities using a tool with predefined disease categories and corresponding weights, it will be helpful to predict patients’ future health status, stratify patients’ risk in advance, and develop potential care management plans.

With the increasing use of health insurance claims databases in health service research, several comorbidity measures based on claims information have emerged in the past several decades. Most of these measures are based on diagnoses assigned to patients in real practice settings, including different versions of the Charlson Index^9,10 and the Elixhauser comorbidity measure.¹¹ Given the limited number of diagnosis codes allowed in each database, diverse diagnosis coding practices among healthcare professionals, and potential coding errors, researchers have attempted to develop pharmacy-based comorbidity measures that use drug dispensing information from health insurance claims databases to measure comorbidity status. The Chronic Disease Score (CDS) is the first reported pharmacy-based comorbidity measure in the U.S.^12,13 The updated CDS was subsequently developed with expanded disease categories.¹⁴ Instead of using disease categories to classify comorbidities, Dong et al used drug categories to describe comorbidities, developed the Pharmacy-Based Disease Index (PBDI), and demonstrated its ability to predict future hospitalization in a Taiwanese adult population.¹⁵ Multiple studies have applied these measures in different populations and examined their performance in predicting mortality, morbidity, and a variety of health utilization.^16–22

Although studies have demonstrated that comorbidities pose negative impacts on the survival of patients with COPD, few studies have applied comorbidity measures driven from health insurance claims database to predict subsequent health outcomes in this population.^23,24 This study aimed to examine the performance of commonly used diagnosis-based and pharmacy-based comorbidity measures to predict future death and hospitalization in patients with COPD. Specifically, we also aimed to explore if the PBDI, which was developed in a Taiwanese adult population, could perform well in a specific disease subgroup.

Methods

Data Sources and Study Population

A single-payer National Health Insurance program was initiated in Taiwan in 1995 and reached an enrollment rate of 99% by 2014.²⁵ The Taiwan National Health Insurance Database comprises comprehensive data of demographic and enrollment records, hospital admissions and outpatient visits, and pharmacy dispensing claims from hospitals, outpatient clinics, and community pharmacies for 99% of the total Taiwanese population (approximately 23 million), which could be linked to the National Death Registry to ascertain death information. All patient and healthcare provider identification codes are de-identified.²⁶ The study protocol was approved by the National Yang-Ming University Research Ethics Committee.

We identified patients with COPD as our study population and defined these patients as those who had at least one inpatient or outpatient COPD diagnosis (International Classification of Diseases, 9th Revision, Clinical Modification [ICD-9-CM] codes 491, 492, or 496) during 2013. The algorithm has been found to have a sensitivity of 85.0% and a specificity of 78.4%.²⁷ We excluded patients with an age less than 40 years or more than 100 years in 2013, patients with ambiguous sex information, and patients who died during years of 2013 or 2014.

Ascertainment of Comorbidity Measures

Among the diagnosis-based comorbidity measures, the Charlson Index includes 17 disease categories that characterize cardiovascular or cerebrovascular, endocrine, respiratory, gastric and hepatological, renal, neurological, oncological, rheumatologic, and miscellaneous comorbidities.^9,10 The Elixhauser comorbidity measure includes 30 disease categories that cover more comprehensive comorbidities of the above systems.¹¹ For example, cardiac arrhythmia and valvular disease are included in the Elixhauser comorbidity measure but not in the Charlson Index. Moreover, the Elixhauser comorbidity measure covers psychological comorbidities such as depression and psychoses. Among the pharmacy-based comorbidity measures, the updated CDS includes 29 disease categories,¹⁴ and the PBDI includes 37 drug categories.¹⁵ Specifically, the PBDI has a unique strength of including “drugs for chronic viral hepatitis”, which quantifies a substantial disease burden in Asia and in Taiwan. Tables S1-S4 show disease or drug categories, diagnosis or drug codes, and the corresponding weights for each comorbidity measure.

For each eligible patient, we used hospital and outpatient diagnoses during the baseline year (i.e., 2014) to construct disease categories and calculate the Deyo score according to the Deyo version of the Charlson Index.^9,10 We also indicated if patients had any of the Elixhauser comorbidities in the same year.¹¹ For the pharmacy-based comorbidity measures, we extracted pharmacy dispensing claims from hospitals, outpatient clinics, and community pharmacies during the same one-year period, constructed each patient’s medication profile, and calculated his or her updated CDS and PBDI based on disease or drug categories and the corresponding weights developed in the original study.^14,15

Ascertainment of Outcomes of Interest

Our outcomes of interest were overall death and overall hospitalization in the subsequent year (i.e., 2015). We also examined if patients were hospitalized due to specific causes, including cardiovascular disease, cerebrovascular disease, or respiratory disease. Table S5 provides detailed information on causes of hospitalization and the corresponding hospitalization diagnosis codes. Patients were categorized as hospitalization due to other reasons if they did not have any of the hospitalization diagnosis codes defined in Table S5.

Statistical Analyses

To compare the predictive performance of each comorbidity measure for subsequent-year death or hospitalization, we randomly divided the eligible patients into a training dataset and a validation dataset of equal size. In the training dataset, we fit separate logistic regression models for each comorbidity and for each outcome of interest. Each comorbidity model included age, sex, plus either the Deyo score, the 30 indicators representing the presence or absence of the 30 Elixhauser comorbidities, the updated CDS, or the PBDI as independent variables and the outcome of interest as the dependent variable. A model with predictors of age and sex only served as the reference model. We calculated the predicted probabilities of outcomes according to the estimated regression coefficients in each model. Furthermore, each model created in the training dataset was applied to the validation dataset. Three analytic methods were used in the training and validation datasets to evaluate how well each comorbidity model could predict subsequent-year outcomes.

First, we computed the c-statistic to evaluate each model’s ability in discriminating patients who did and patients who did not experience death or hospitalization in the subsequent year.²⁸ Second, we computed the net reclassification improvement (NRI) to evaluate if the individual comorbidity models could reclassify more patients experiencing outcomes into a higher risk stratum and more patients not experiencing outcomes into a lower risk stratum than the reference model. Specifically, we classified patients into different risk strata (i.e., very low, low, intermediate, and high) based on their predicted probabilities of experiencing outcomes derived by each model. Among patients with and without subsequent-year outcomes, we constructed reclassification tables by comparing the number of patients in each risk stratum of each comorbidity model with the reference model, respectively. Based on the reclassification tables, we estimated the NRI as [P(up|D=1)-P(down|D=1)] + [P(down|D=0)-P(up|D=0)], where D=1 if an individual experiences any outcome of interest in the subsequent year and D=0 otherwise; the upward movement (up) indicates an individual was reclassified into a higher risk stratum by the comorbidity model versus the reference model, and the downward movement (down) indicates reclassification in the opposite direction. Therefore, the NRI represented the proportion of individuals that were correctly reclassified, with a positive value suggesting that the comorbidity model can classify more individuals into the correct risk stratum than the reference model.²⁹ Third, to evaluate the calibration of each model, we grouped patients into deciles according to the predicted probabilities of experiencing outcomes in each model. We compared the predicted probability and the observed incidence of subsequent-year outcomes within each decile.²⁸

We conducted additional analyses to examine the robustness of model performance. First, we restricted the study population to those who had at least one inpatient or three outpatient COPD diagnoses during 2013 to minimize potential misclassification of the COPD population. In addition, we excluded those who died in 2015 in the model that predicted subsequent-year hospitalization to reduce the misclassification of patients who died as alive and without hospitalization.

All statistical analyses were performed using SAS version 9.4 (SAS Institute, Cary, NC).

Results

Eligible Patients

We identified a total of 428,251 eligible patients (Figure S1). The mean (standard deviation [SD]) age of the cohort was 67 (13) years, and 60% of the patients were male. The mean (SD) number of disease or drug categories ranged from 1.7 (1.4) to 5.9 (3.6) for different comorbidity measures (Table 1). The most commonly identified comorbidities included hypertension, anxiety and tension, gastric acid disorder, and pain and inflammation. The most commonly used non-respiratory medications included angiotensin converting enzyme inhibitors or angiotensin II receptor blockers, calcium channel blockers, glucocorticosteroids, anxiolytics, drugs for peptic ulcer or gastroesophageal reflux disease, and nonsteroidal anti-inflammatory drugs.

Table 1 Patient Characteristics of the Study Cohort

During the subsequent year, approximately 6% and 27% of study patients died and were hospitalized, respectively. Most of the patients were hospitalized due to respiratory disease (15%), followed by cardiovascular disease (7%). Patients in the training dataset and patients in the validation dataset showed similar demographic characteristics, distribution of comorbidity measures, and probabilities of death or hospitalization during the subsequent year (Table 1).

Predictive Performance of Different Comorbidity Models

The c-statistics for subsequent-year health outcomes with different comorbidity models are listed in Table 2. For overall death in the validation dataset, the reference model with predictors of age and sex only yielded a c-statistic of 0.768; however, all the models additionally including different comorbidity measures reached c-statistics greater than 0.8. The Elixhauser comorbidity measure performed best (c-statistic=0.832), followed by the PBDI (c-statistic=0.822), the Deyo version of the Charlson Index (c-statistic=0.815), and the updated CDS (c-statistic=0.808). For overall hospitalization, the model discrimination tended to be lower but still showed c-statistics greater than 0.7. The PBDI and the Elixhauser comorbidity measure performed best (c-statistics of 0.730 and 0.724, respectively), followed by the updated CDS and the Deyo version of the Charlson Index (c-statistics of 0.714 and 0.710, respectively). In terms of cause-specific hospitalization, the Elixhauser comorbidity measure and the PBDI still had greater discrimination than the updated CDS and the Deyo version of the Charlson Index. Of interest, each comorbidity measure predicted hospitalization due to cardiovascular disease, respiratory disease, or cerebrovascular disease better than overall hospitalization. For example, for hospitalization due to cardiovascular disease and respiratory disease, the Elixhauser comorbidity measure had c-statistics of 0.792 and 0.787, and the PBDI had c-statistics of 0.776 and 0.769. However, none of the models optimally predicted hospitalization due to other reasons, with c-statistics lower than 0.6.

Table 2 C-Statistics of Different Comorbidity Measures in Relation to Subsequent-Year Health Outcomes

All the NRI values in Table 3 are positive, which indicates the improved reclassification ability of each comorbidity model versus the reference model for the subsequent-year outcomes. In general, the reclassification abilities of the Elixhauser comorbidity measure and the PBDI remained better than the Deyo version of the Charlson Index and the updated CDS for different outcomes. For example, for overall death and in the validation dataset, approximately 26%, 24%, 21%, and 15% of individuals were correctly reclassified by the Elixhauser comorbidity measure, the PBDI, the Deyo version of the Charlson Index, and the updated CDS, respectively. The cut-off points of probabilities of each outcome for different risk strata are shown in Table S6.

Table 3 Net Reclassification Improvement of Different Comorbidity Measures in Relation to Subsequent-Year Health Outcomes

Figure 1 presents the observed and predicted probabilities of each outcome in each decile defined by the predicted probabilities of outcomes of interest derived by the PBDI. The observed probabilities of overall death, overall hospitalization, and cause-specific hospitalization increased from the lowest risk decile to the highest risk decile and closely matched the corresponding predicted probabilities. However, the observed and predicted probabilities of hospitalization due to other reasons did not greatly vary among risk deciles, which suggested poor calibration abilities. The findings derived by other comorbidity models show similar patterns (data not shown).

Figure 1 Observed and predicted probabilities of subsequent-year outcomes among individuals in deciles defined by the predicted probability of the outcome defined by the PBDI in the validation dataset (n=214,125). Abbreviations: CV, cardiovascular; PBDI, pharmacy-based disease indicator.

Results of Additional Analyses

When we restricted the study population to those who had at least one inpatient or three outpatient COPD diagnoses during 2013 or those who did not die in 2015, the model discrimination of the Elixhauser comorbidity measure and the PBDI were better than that of the Deyo version of the Charlson Index and the updated CDS for different outcomes, although the c-statistics tended to be lower than that in the main analysis (Table S7 and Table S8).

Discussion

This population-based study examined the predictive performance of a variety of comorbidity measures derived from health insurance claims for health outcomes in patients with COPD. The Elixhauser comorbidity measure performed best, followed by the PBDI, and in general, the Deyo version of the Charlson Index and the updated CDS showed similar predictive abilities for health outcomes. In terms of individual outcomes, all comorbidity measures had excellent discrimination in predicting overall death (c-statistics > 0.8) and acceptable discrimination in predicting overall hospitalization (c-statistics > 0.7). Moreover, each comorbidity measure predicted hospitalization due to cardiovascular disease, respiratory disease, or cerebrovascular disease better than overall hospitalization but did not adequately predict hospitalization due to other reasons. The performance ranks did not greatly differ for different predictive statistics or definitions of COPD or when we restricted the study population to those who did not die in the subsequent year.

Comparison of Different Comorbidity Measures

As the use of health insurance claims databases in healthcare settings is increasing, our results provide important implications in terms of describing the health status of patients with COPD using health insurance claims information. Our study indicated that the Elixhauser comorbidity measure performed best in predicting any outcome of interest in the subsequent year. Chu et al found the Elixhauser comorbidity measure yielded a higher c-statistic (0.701) than that of the Charlson Index (0.681) in predicting subsequent-year death among patients with COPD,²³ although the difference for measures was minimal in another study conducted by Austin et al (c-statistics: 0.822 for the Elixhauser comorbidity measure and 0.819 for the Charlson Index).²⁴ Quail et al also found that the Elixhauser comorbidity measure had superior performance in predicting overall death and overall hospitalization in a general population, an elderly population, patients with diabetes, and patients with osteoporosis.¹⁸

Compared to other tested comorbidity measures, the Elixhauser comorbidity measure includes more comprehensive disease categories across organ systems of the human body. The weights were derived specifically for individual health outcomes in the current cohort. For example, for overall death and overall hospitalization, the categories of metastatic cancer and alcohol abuse had the highest weights. For cause-specific hospitalization, cardiovascular comorbidities (congestive heart failure and cardiac arrhythmias), cerebrovascular comorbidities (paralysis and other neurological disorders), and respiratory comorbidities (chronic pulmonary disease) played the most influential roles (Table S9). Therefore, it is not surprising that he Elixhauser comorbidity measure showed the best performance. However, it requires a sufficient sample size, and it may be tedious to re-derive weights for the 30 Elixhauser comorbidity categories for the outcome of interest and population each time it is used. Therefore, researchers need to consider the available sample size and to what extent they would like to achieve the predictive performance when selecting candidate comorbidity measures for their target population.

Our study showed the PBDI performed right next to the Elixhauser comorbidity measure but better than the Deyo version of the Charlson Index and the updated CDS. Several reasons may explain these findings. First, compared to the updated CDS, the PBDI includes more drug categories. In addition, the PBDI applies the weights derived from a general Taiwanese population. Third, unlike many health insurance claims databases which do not have information on drug use during hospitalization,³⁰ data on inpatient drug use are available in Taiwan’s NHIRD.²⁶ These factors all facilitate the PBDI to reveal more comprehensive comorbidities and to better predict health outcomes.

Dong et al indicated that the PBDI performs numerically better than the Deyo version of the Charlson Index in predicting the subsequent-year overall hospitalization in a general Taiwanese population (c-statistics of 0.715 and 0.691, respectively).¹⁵ The current study further demonstrated satisfactory performance of the PBDI in predicting various health outcomes in patients with COPD. If we treat 37 PBDI drug categories, rather than a single PBDI score, as independent variables and apply the weights derived for each drug category, the PBDI can perform even better than the Elixhauser comorbidity measure (Table S10). Compared to Chu and Austin’s studies, which only examined the predictive performance of diagnosis-based comorbidity measures in patients with COPD,^23,24 our study provides a more complete comparison across diagnosis-based and pharmacy-based comorbidity measures. Our findings suggest that the PBDI, which is based on drug dispensing information, is useful to stratify patient risk and may facilitate the development of care management plans in real practice. Further research can be conducted to examine its predictive ability in other disease populations or in other countries.

Comparison of Different Health Outcomes

Our study showed each comorbidity model better predicted overall death than overall hospitalization, which is consistent with what has been observed in other studies.^16–19,21 Compared to previous findings, our study further demonstrated that each comorbidity model can still achieve satisfactory discrimination when we focused on hospitalization due to cardiovascular disease, cerebrovascular disease, or respiratory disease, which are the leading causes of hospitalization in the COPD population.^31–33 This observation may be because compared to overall death and the leading causes of hospitalization, patients hospitalized due to miscellaneous reasons are susceptible to non-disease factors that could not be captured by the comorbidity measures. These results also suggest that researchers target disease-specific outcomes when applying these comorbidity measures in COPD management.

Limitations

Our study has several limitations. First, we focused on comorbidity measures that use diagnosis and pharmacy dispensing claims data. Similar to most claims databases, the Taiwan health insurance claims database does not contain information on clinical parameters, such as the forced expiratory volume in one second. Therefore, we could not compare the predictive performance of the claims-based comorbidity measures with that of available multi-dimensional indexes (such as the BODE and CODEX indexes) that are highly associated with COPD prognosis and require information on many clinical parameters.^34,35 In addition, we did not examine the predictive abilities of the well-known Johns Hopkins Adjusted Clinical Group System because it requires computation with complex and commercial algorithms.³⁶

Conclusion

In conclusion, our study comprehensively compared the predictive performance of various comorbidity measures and various health outcomes in the setting of a population-based health insurance claims database. The Elixhauser comorbidity measure and the PBDI are useful tools for describing comorbid conditions and predicting health outcomes in patients with COPD.