Development and Validation of a Healthcare Utilization-Based Algorithm to Identify Acute Exacerbations of Chronic Obstructive Pulmonary Disease

Plain Language Summary

Acute exacerbations of chronic obstructive pulmonary disease (AECOPD events) are a sudden worsening of a patient’s respiratory symptoms that is beyond normal day-to-day variations and leads to a change in medication. However, there remains no clear consensus on how best to identify these exacerbations using administrative data, such as electronic records of medical claims. This is an important issue, as accurate identification of AECOPD events in electronic administrative data is essential for improving population health surveillance and practice management.

We developed two algorithms that could be applied to administrative data; one of these algorithms aimed to identify moderate AECOPD events, while the other aimed to identify severe AECOPD events. In this study spanning between January 1, 2010 and September 30, 2015, we used these newly developed algorithms to identify moderate and severe AECOPD events within two large regional integrated healthcare systems. We then assessed the performance of the algorithms by calculating both their positive and negative predictive values (PPV and NPV). These statistical values represent the probability of a positive or negative identification being accurate. To calculate predictive values, the results of each algorithm were compared with AECOPD events identified in medical records – a process that was adjudicated by an expert in the field.

Our results show that, in terms of both PPV and NPV, the developed algorithms can identify moderate and severe AECOPD events in administrative databases with sufficient validity. The proposed algorithms could therefore prove beneficial in comparative effectiveness research, health plan quality improvement projects, and practice management.

Introduction

Chronic obstructive pulmonary disease (COPD), a condition characterized by persistent airflow limitation and associated with enhanced inflammatory response of the lungs to irritants,¹ is a significant burden to both individuals and healthcare systems. The overall prevalence of COPD among US adults is estimated to be 6.2%² and is approximately 12% among Medicare beneficiaries.³

The Global Initiative for Chronic Obstructive Lung Disease (GOLD) committee defines an acute exacerbation of COPD (AECOPD) as “an acute worsening of respiratory symptoms that results in additional therapy.”⁴ Exacerbations are often classified by severity: typically, “moderate” exacerbations are defined as those that require a COPD-related emergency department (ED) visit or outpatient visit with systemic corticosteroid treatment and/or antibiotic therapy, whilst “severe” exacerbations are those requiring hospitalization.⁴ Exacerbations are associated with a significant increase in mortality and healthcare resource utilization in the form of hospitalizations, ED visits, urgent care visits, and prescription medications.^5–7 Patients suffering from more frequent exacerbations can experience acute irreversible pulmonary function loss and reduced quality of life.^8–11 Preventing exacerbations is therefore a primary goal in COPD treatment.¹

Administrative data – computerized systems of healthcare events where healthcare utilization is summarized using standardized codes (eg, diagnosis, procedure, prescriptions) – are the predominant source of real-world healthcare data in the US and are used for health services, as well as pharmacological and epidemiological studies of COPD. These systems include claims data for fee-for-service billing and encounter data for managed care entities receiving capitated payments. While limited to insured individuals who access healthcare,^12,13 administrative data provide valuable information about the healthcare utilization experience of individuals with COPD and real-world management. The notion of AECOPD as a worsening of underlying COPD has its underpinnings in changes in healthcare utilization;^14,15 however, there remains no consensus on how best to identify exacerbations using administrative data.^14,16 This lack of consensus, across the range of mild to severe, represents a significant research barrier^16–18 for a retrospective analysis of large healthcare databases where detailed clinical outcomes of the study population are not always available as indicators of exacerbation events. The capability of large databases to identify patients who may benefit from COPD interventions depends on correct identification of AECOPD. Practical methods are needed to enable this identification to be performed with confidence.¹⁹

The aims of this study were to develop two utilization-based algorithms that could be applied to administrative data to identify moderate and severe AECOPD and to validate their performance.

Materials and Methods

Study Design

This was a retrospective longitudinal cohort study spanning January 1, 2010 – September 30, 2015. The index date was defined as the first prescription of GOLD¹ guideline-recommended therapy for COPD (Supplementary Table 1), during or following at least one hospitalization (minimum length of stay ≥2 days), one ED visit, or two outpatient visits, associated with COPD diagnoses codes as identified from utilization data. There was a baseline period of 12 months prior to the index date during which inclusion and exclusion criteria were applied (Figure 1). A patient’s observation period began with the index date and ended with the earliest date of death, health plan disenrollment, or end of study period. Potential AECOPD events were identified during the observation period using diagnosis codes from the International Classification of Disease, Ninth Revision, Clinical Modification (ICD-9-CM).

Figure 1 Study design. Note: This diagram is not to scale.Abbreviations: COPD, chronic obstructive pulmonary disease; GOLD, Global Initiative for Chronic Obstructive Lung Disease; Q, quarter.

Study Population

Patients aged ≥40 years with ≥1 hospitalization, ≥1 ED visit, or ≥2 outpatient visits with a primary or secondary COPD diagnosis (ICD-9-CM codes: 491.xx, 492.xx, and 496) during the 12-month baseline period were included in the study. Additional inclusion criteria were: ≥1 prescription for any GOLD guideline-recommended maintenance therapy for COPD (stand-alone or in combination); ≥12 months of continuous health insurance coverage prior to the index date; ≥1 month of continuous health insurance coverage after the index date; and availability of electronic medical records (EMR; inclusive of claims for payment from contract healthcare providers) data. Patients were excluded if they had any of the following comorbid conditions during baseline: active pulmonary tuberculosis, sarcoidosis, respiratory cancer, cystic fibrosis, extrinsic allergic alveolitis, pneumoconiosis, pulmonary congestion and hypostasis, other alveolar and parietoalveolar pneumonopathy, lung involvement in conditions classified elsewhere, pulmonary collapse, pulmonary eosinophilia, or allergic bronchopulmonary aspergillosis.

Data Sources

Health insurance encounters and EMR data were obtained from two independent health systems: Kaiser Permanente Mid-Atlantic States (KPMAS) and Reliant Medical Group, Inc. (Reliant). KPMAS is an integrated health system, comprised of over 1600 providers practicing in 33 medical centers in Baltimore, Washington, D.C., Northern Virginia, and Southern Maryland, USA, serving more than 770,000 members. Reliant is the largest private, multi-specialty group in central Massachusetts, USA, providing comprehensive care for more than one million patient visits a year with more than 300 providers practicing in over 20 locations.

Study Outcomes

Baseline Characteristics and Algorithm Development

Baseline characteristics, including patient demographics and medical history, were collected for patients meeting inclusion criteria. In addition, the prevalence of diagnosis and procedure codes, as well as healthcare utilization data commonly used to define COPD and AECOPD events, were tabulated for the observation period. Based on the research team’s expertise in data related to COPD, previously published algorithms^19–25 were modified to develop separate algorithms to identify “moderate” and “severe” AECOPD events in administrative data.

Algorithm Performance by Positive Predictive Value

Of the potential AECOPD events identified with the utilization-based algorithms, random samples of 550 (300 moderate and 250 severe) were selected for detailed medical chart review to validate whether the potential AECOPD was a true event. Validation of event identification and classification was based on review and abstraction of computerized medical records by trained research associates at KPMAS and experienced pulmonary nurses at Reliant.

A standardized abstraction instrument was designed to systematically capture COPD symptom changes, treatment changes and other utilization associated with the event, and spirometry and other clinical data supporting the diagnosis of COPD (Supplementary Table 2). Definitions of validated (true positive) AECOPD events were based on the GOLD COPD 2017 description of AECOPD and associated utilization. An event was classified as “definite” AECOPD if it met both major criteria: 1) acute worsening of COPD symptoms; and 2) a change in treatment such as a new prescription of antibiotics and/or corticosteroids or increase in respiratory medications. An event was classified as “probable” AECOPD if it met one of the major criteria, plus ≥4 minor criteria associated with utilization for AECOPD events (Supplementary Table 3). An event was classified as “possible” AECOPD if it only met ≥4 of the minor criteria.

Events were adjudicated by a pulmonologist with experience in COPD studies. EMR abstractors reviewed events that did not meet the definition of “definite” AECOPD events to identify potential reasons they were not fully confirmed. Positive predictive value (PPV) was computed using the true and false positives (where the unit of analysis was the AECOPD event) initially identified by medical record review and abstraction and then adjudicated by the pulmonologist.

Spirometry data were collected whenever available to further confirm validity of the COPD diagnosis. If multiple tests were performed, the one closest to the AECOPD event was selected. “COPD verified” patients had: a ratio of forced expiratory volume over 1 second (FEV₁) to forced vital capacity (FVC) of <0.70; “COPD unverified” patients had an FEV₁/FVC ratio ≥0.70 (these patients typically had reduced percent predicted FEV₁ and FVC); spirometry results could not be found for “COPD undefined” patients.

Algorithm Performance by Negative Predictive Value

Random samples of 100 patients who were identified by the algorithms as having no AECOPD during the observation period were selected to assess negative predictive value (NPV). Medical records were reviewed to find one potential moderate or severe event missed by the algorithm. If found, the AECOPD event was abstracted using the standardized form and adjudicated to determine if a true event had been missed. NPV was then calculated, with patients as the unit of analysis.

Results

Study Population

The sample selection flowcharts for Reliant and KPMAS data are presented in Figure 2. After applying inclusion and exclusion criteria, 2366 patients from Reliant and 5548 patients from KPMAS were selected.

Figure 2 Patient flow diagrams. Abbreviations: COPD, chronic obstructive pulmonary disease; ED, emergency department; EMR, electronic medical record; KPMAS, Kaiser Permanente Mid-Atlantic States; Reliant, Reliant Medical Group, Inc.

Baseline Characteristics and Algorithm Development

Patient baseline characteristics are presented in Table 1. Patients were continuously enrolled in their healthcare plan for an average of 2.6 years following the index date at Reliant and 2.0 years at KPMAS. At both sites, the most common (≥20% of patients) medication classes initiated on the index date were short-acting β2 agonists (SABA), inhaled corticosteroids (ICS)/long-acting β2 agonists, and short-acting muscarinic antagonists.

Table 1 Baseline Characteristics at Index Date

Table 2 displays the frequencies of healthcare utilization and respiratory conditions during baseline. The proportions of patients who were hospitalized ≥2 days were very similar (Reliant: 24.0%, KPMAS: 24.6%). Between the two sites, however, utilization for ED visits, observation visits, and short hospital stays (length of stay = 1 day) varied greatly, presumably representing differences in how intermediate-level services were classified or delivered. Most patients had at least one outpatient visit (Reliant: 94.2%, KPMAS: 98.8%) and use of other services (Reliant: 96.4%, KPMAS: 97.3%) during baseline. The most common respiratory condition at baseline was “Chronic airway obstruction, not elsewhere classified (496)” (Reliant: 92.4%, KPMAS: 85.7%).

Table 2 Healthcare Utilization and Clinical Characteristics in the Baseline Period

The final two algorithms contained utilization-based information deemed clinically relevant in identifying patients with moderate or severe exacerbations (Table 3). For example, key components of the algorithm used to identify moderate events were ≥1 ED or outpatient visit relating to respiratory comorbidities or symptoms, along with ≥1 prescription of antibiotics or systemic steroid medications. For the algorithm to identify severe events, key components included ≥1 hospitalization relating to respiratory comorbidities or symptoms.

Table 3 Algorithms for Moderate and Severe AECOPD Exacerbations Used in Round 1

Common Diagnoses and Use of Services During Observation Period

The frequencies of healthcare utilization and respiratory conditions based on diagnosis codes during the period in which the algorithms were applied are presented in Table 4. Most patients had ≥1 outpatient visit (Reliant: 92.9%, KPMAS: 96.5%) and used other services (Reliant: 96.9%, KPMAS: 95.7%) during observation. Utilization related to acute events (ED visits and hospitalizations) was higher in the observation period (vs the baseline), and correspondingly more patients received the diagnoses summarized in Table 4.

Table 4 Healthcare Utilization and Clinical Characteristics in the Observation Period

Validation of Algorithm Using Positive Predictive Value

Validation results for moderate and severe AECOPD algorithms are presented in Tables 5 and 6. Among the moderate AECOPD events, 98.3% (293 of 298) identified by the algorithm were true positives (Table 5). Among the 225 suspected severe exacerbations identified by the algorithm, 216 (96.0%) were true positives (Table 6). The PPV (95% confidence interval [CI]) for severe exacerbations was lowest for the diagnosis “Chronic airway obstruction, not elsewhere classified” (PPV: 94.0% [95% CI: 86.7–98.0]). When evaluated by COPD status (verified, unverified, or undefined, dependent on spirometry results), the PPV was ≥93.8% across all categories for both the moderate and severe algorithms (Supplementary Tables 4 and 5, respectively).

Table 5 Validation Results for Moderate COPD Exacerbation Algorithm

Table 6 Validation Results for Severe COPD Exacerbation Algorithm

Validation of Algorithm Using Negative Predictive Value

Performance of the algorithm was assessed by calculating the NPV (Table 7). NPV was lower for moderate exacerbations compared with severe exacerbations (75.0% [95% CI: 65.3–83.1] vs 95.0% [95% CI: 88.7–98.4], respectively). The “moderate” NPV for Reliant was 72.0% (95% CI: 57.5–83.8) and for KPMAS, 78.0% (95% CI: 64.0–88.5). The “severe” NPV value was 90.0% (95% CI: 78.2–96.7) for Reliant and 100% (95% CI: 92.9–100.0) for KPMAS.

Table 7 Performance of Algorithms by Calculating NPV

Discussion

This study demonstrates that structured, computerized algorithms can identify moderate or severe AECOPD events in administrative databases with sufficient validity for use in health outcomes research and quality improvement projects.²⁶ We also show that the algorithms work equally well for COPD patients who had documented airflow obstruction on pulmonary function tests (PFTs), and those who did not have PFT results available or those who had a FEV₁/FVC ratio >0.70.

The final algorithms build on previously published algorithms and are based on coded values deemed clinically relevant in identifying patients with moderate or severe exacerbations.^19–25 As part of the process for our algorithm development, we incorporated a novel system for validating AECOPD events. Validation was based on the concept that an exacerbation is an acute worsening of respiratory symptoms that results in a change of treatment (major criteria), but also considers clinical factors associated with AECOPD events (minor criteria), such as signs of COPD on radiographs or PFTs. Healthcare utilization such as sputum cultures and blood gases (construct validity) are often associated with AECOPD events, and provider documentation verified that the event was an AECOPD episode and not some other disease process such as heart failure (clinical validation).

Our algorithms build on work from previous studies that examined the validity of using ICD-9 codes to identify AECOPD events in healthcare administrative databases. Stein et al²⁷ examined the performance characteristics of four different algorithms, using ICD-9 coding, to identify AECOPD hospitalization events at two urban academic medical centers. The study used a stratified probability sample of 200 hospitalizations with and without COPD. Interestingly, they found that the algorithms’ PPV ranged from 81.2% to 97.2%, but the algorithm that was most comprehensive and similar to ours was at the low end of this range, while our PPV for severe events was 96.0%. The NPV for their severe algorithms ranged from 93.0% to 93.9%, similar to our 95.0%. Oelsner et al²⁸ also used ICD-9 coding to identify 216 patients who had likely “Chronic Lower Respiratory Disease” events, which encompassed exacerbations of COPD, chronic bronchitis, emphysema, or asthma that resulted in ED visits or hospitalizations. As with our study, they established a priori criteria for “probable”, “highly probable”, and “definite” events, with adjudication by physician review.²⁸ They found that the order in which the discharge diagnoses were listed had importance on the validity of the event; for example, the PPV of a “Chronic Lower Respiratory Disease” exacerbation resulting in hospitalization was >95% if a respiratory disease was the primary discharge diagnosis, but <55% if a secondary diagnosis. Stanford et al recently summarized results for a similar exacerbation algorithm effort.²⁹ Their algorithm used the Stein et al algorithm as a starting point. They did not separate moderate and severe exacerbations, but instead developed a combined algorithm. They reported a lower PPV value (67.5%) than found in our study. However, while our algorithm only considered events for which diagnosis codes were a primary diagnosis, the Stanford et al algorithm included events for which diagnosis codes may have been a primary or secondary diagnosis. Our study adds to current literature by demonstrating the validity of using healthcare utilization captured in large administrative databases, further outlining objective criteria that can be used to define and confirm AECOPD events.

Replication and consistency of the PPV results between sites is a strength of this study, bolstering external validity. The lower NPV values were indicative of the broader range of events encompassed by moderate exacerbations – from borderline mild to borderline severe – and the broader range of settings in which these events could be documented. Exacerbations missed by the algorithms in this study were reviewed using administrative and EMR data to understand potential reasons for omission. Reasons included: missing claims (ie, medications or visits not appearing in administrative data), timing (eg, medication dispensed before the medical visit, which the algorithm did not consider as being associated with the visit), or because the algorithm did not consider secondary diagnoses as indications of AECOPD. Missed moderate exacerbations often occurred near the index date, which was based on the first prescription fill for a COPD maintenance medication; if a respiratory problem was not listed as the primary diagnosis, then these events were missed. These observations suggest potential improvements that may be made to improve the overall performance of this classification system.

Another reason for the relatively low NPV in moderate exacerbations is the location of care (ie, home health services and telehealth services). For instance, Reliant has a COPD disease management group that tries to manage mild/moderate exacerbations via telephone or in person. Many patients treated by this group have pre-filled medications they can begin taking at the onset of symptoms. These patients are instructed to call their disease management nurse when initiating this treatment so that this can be confirmed and documented within the EMR. Services like these are becoming more prevalent, meaning that claims-based algorithms that do not include them have the potential to miss moderate AECOPD events.

PPV was calculated among patients with verified, unverified, and undefined COPD as per spirometry results. Results were robust to the main analysis and the PPV remained high across all three groups, suggesting the algorithm performs well without the need for spirometry results. Lack of PFT data in most administrative claims databases is frequently cited as a weakness, but this analysis suggests that lack has little appreciable impact on algorithm performance. It could be that the high use of PFTs and chest computed tomography scans in these two systems, compared to previous studies, has resulted in better defined research populations with fewer misclassifications.

Several limitations should be noted. EMRs are organized to collect and present data for clinical support, not necessarily research. Our algorithm was applied to structured codified data as is common in claims and encounter data. Different healthcare systems may prioritize electronic data collection for different purposes; therefore, algorithm performance may differ in other systems. Another limitation is that PPV and NPV were the only measures assessing algorithm performance. However, the sampling required to assess additional metrics (eg, sensitivity, c-statistic, net reclassification improvement, or the incremental discrimination index) were beyond the scope of the study. While the algorithms were robust – being validated in two different healthcare systems – it is not known whether the results can be generalized to more limited administrative databases in the US or elsewhere in the world. ICD-10-CM coding was not in use during the study time period, which could limit the generalizability of the algorithms for future use. However, though not validated, the authors have created a preliminary crosswalk from ICD-9 to ICD-10 to address this point (Table 3 and Supplementary Tables 6 and 7). Underlying concepts that have been validated remain constant between ICD-9 and ICD-10, and the severe AECOPD codes listed in Supplementary Table 7 closely align with codes used for the CMS COPD Readmission Measure.³⁰ Finally, since there is no specific ICD-9 diagnosis code for COPD, patients were identified based on COPD-associated diagnoses and GOLD guideline-recommended maintenance therapy. This may have led to the misclassification of patients into the COPD cohort. However, the high PPV observed across all diagnoses (including “496 - Chronic airway obstruction, not elsewhere classified”) and the relatively low and expected proportion of patients with concomitant asthma during the observation period indicate that misclassification was unlikely to have affected our conclusions.³¹

Conclusions

In this study, we developed and validated, through medical record review, improved utilization-based algorithms to identify moderate and severe AECOPD in administrative databases at two sites. PPV for both algorithms was very high; NPV was lower for the moderate exacerbation algorithm. Our proposed algorithms could prove useful for comparative effectiveness research, health plan quality improvement projects, practice management, and to increase knowledge on the frequency of COPD exacerbations using large administrative databases. Opportunities to improve our algorithms – such as inclusion of home health services and clinical data (radiology, laboratory, spirometry) which is increasingly being incorporated into computerized databases – should be explored.