Development and Validation of a Multivariable Prediction Model to Identify Acute Exacerbation of COPD and Its Severity for COPD Management in China (DETECT Study): A Multicenter, Observational, Cross-Sectional Study

Plain Language Summary

Chronic obstructive pulmonary disease causes breathing-related problems and airflow blockage. As the disease progresses, patients often experience worsening of symptoms in a process called exacerbation. Exacerbation can lead to poorer quality of life and even hospitalization. Physicians can better manage the disease if exacerbations are detected early. However, it is difficult for physicians to always detect exacerbations early because hospitals and medical centres sometimes lack accurate and objective measurement tools.

In this study, Yin et al developed two measurement tools to help physicians detect and manage exacerbations in their patients with COPD. The first tool indicates whether a patient is experiencing an exacerbation, using information on disease-related history, and current signs and symptoms. The second tool indicates whether the patient has a severe exacerbation, using information on previous exacerbations, as well as current disease-related symptoms.

Physicians can better manage chronic obstructive pulmonary disease by using these measurement tools to detect exacerbations early on, which can help to prevent substantial worsening of the disease.

Introduction

Chronic obstructive pulmonary disease (COPD), characterized by persistent airflow limitation and chronic respiratory symptoms, is one of the leading causes of morbidity and mortality globally.¹ In China, COPD affects close to 100 million individuals (13.6% of adults aged ≥40 years),² and was identified as one of the top three causes of disability-adjusted life-years in 2017.³ COPD exacerbations are part of the natural course of the disease and are defined as an acute worsening of respiratory symptoms that result in a need for additional therapy.⁴ A meta-analysis of seven randomized controlled trials comprising 18,746 patients reported nearly one-quarter of patients with COPD experienced ≥1 severe exacerbation in the previous year;⁵ in China, patients with COPD experience 0.5–3.5 episodes of an acute exacerbation of COPD (AECOPD) annually.⁶ Acute exacerbations may enhance disease progression and are major drivers of economic burden. Among patients in China who were hospitalized for AECOPD, long-term mortality was 37.2%.⁷ The mean per capita expenditure for AECOPD increased from 3,472 USD to 4,538 USD between 2009 and 2017.⁸ Given that there is a limited window of time between exacerbation onset and possible hospitalization, early detection and intervention for AECOPD are crucial to improve disease prognosis.⁹

There are several challenges in detecting AECOPD, particularly in primary healthcare settings in China, leading to frequent misdiagnoses and missed diagnoses. These include: 1) there exists a lack of objective measurements, and diagnosis relies largely on clinical symptoms and the types of healthcare resources used;¹⁰ 2) spirometry is currently the most reproducible and objective method of measuring airflow limitation,¹ but many community hospitals in China lack access to spirometry equipment;¹¹ 3) comorbidities are common in patients with COPD,⁶ which may enhance the challenge of differential diagnosis; 4) there is a low level of COPD-specific knowledge among physicians in primary hospitals;¹² and 5) patients may lack COPD awareness and a formal COPD diagnosis.²

There is a clear unmet clinical need for an accurate and objective diagnostic tool for early detection of AECOPD. Most existing models are predictive of future exacerbations, but do not detect current exacerbations.¹³ One diagnostic tool that captures exacerbation frequency, severity and duration using a patient-reported daily symptom diary is the exacerbations of chronic pulmonary disease tool (EXACT); however, this tool fails to detect up to two-thirds of clinically important outpatient COPD exacerbations.¹⁴ To address this gap, the DETECT study (NCT03556475) aimed to develop and validate multivariable diagnostic models to identify AECOPD in patients diagnosed with COPD and non-COPD patients who present with respiratory symptoms (primary objective), and severe AECOPD in patients diagnosed with AECOPD (secondary objective) in community hospitals across China. Development of an AECOPD identification tool using independent variables that are easy to obtain in the clinic setting will help to address the challenges currently faced in China’s primary healthcare settings, as well as the urgent global need for early AECOPD detection and improved disease intervention.¹

As a prerequisite to the study, a literature search (to December 31, 2016) was conducted through PubMed, Embase, Ovid, Science Direct and Web of Science. Search terms included (“COPD” or “chronic obstructive pulmonary diseases” or “AECOPD”) AND (“diagnosis” or “diagnostic” or “screen”), and the search field was limited to the article title. Of 523 studies screened, 108 full-text study publications were accessed and reviewed for potential candidate variables to use in development of the multivariable prediction models. These variables were reviewed and verified by respiratory experts in China before application to the present study.

Methods

Study Design and Population

DETECT was an observational, multicenter, cross-sectional study conducted in seven tertiary hospitals with respiratory experts across six regions of China (Eastern, Southern, Northern, Central, Western and Northeastern regions). Patients were consecutively enrolled between April 22, 2020 and January 18, 2021. Key eligibility criteria were: 1) ≥40 years of age; 2) diagnosed with moderate/severe stable COPD (with a diagnosis of COPD, predicted ratio of forced expiratory volume in 1 second [FEV₁]/forced vital capacity [FVC]< 0.7, and FEV₁< 0.8 predicted), AECOPD (with a diagnosis of COPD, predicted ratio of FEV₁/FVC< 0.7, and consulted the outpatient clinic or emergency department due to acute worsening of respiratory symptoms with an AECOPD diagnosed by more than two clinical experts), or no COPD (with chronic respiratory symptoms including cough, sputum, wheeze, and dyspnea, and ≥2 high-risk factors including symptoms, exposure, health history, and recent history of respiratory event) per Global Initiative for Chronic Obstructive Lung Disease (GOLD) 2017 criteria.¹⁵ Full inclusion and exclusion criteria can be found in Table S1. COPD exacerbation was defined as worsening of ≥2 major symptoms or 1 major and ≥1 minor symptom for ≥2 consecutive days (or 1 day for very rapid symptom development); major symptoms include dyspnea, increase in sputum volume, and increase in sputum color/purulence, and minor symptoms include sore throat, nasal discharge and/or congestion, fever without other cause, increased cough, and increased wheeze. Mild exacerbations were defined as having symptoms that did not require hospitalization, systemic steroids, or antibiotics for ≥3 days; moderate exacerbations were defined as having symptoms that required systemic corticosteroids and/or antibiotics for ≥3 days but did not require hospitalization or >24 hours in the emergency department; severe exacerbations were defined as having symptoms that required hospitalization (including >24 hours in the emergency department).

Considering a 20% margin for incomplete data and estimated two-sided 95% confidence interval (CI) widths of 0.191 and 0.155 that were based on sensitivity and specificity of 0.70 for internal validation, we aimed to enroll 300 patients, including 90 non-COPD patients with high-risk factors, 90 moderate-to-severe COPD patients, and 120 AECOPD patients (60 mild/moderate, and 60 severe). At a face-to-face visit, information on patient demographics, signs and symptoms, disease and medication history, and health status were collected. Laboratory and spirometry tests were not mandatory, but data were collected if available. The data obtained by respiratory physicians were collected through case report forms, medical records, and patient-reported outcome (PRO) questionnaires. The study was designed and conducted in accordance with the Declaration of Helsinki, International Conference on Harmonization Good Clinical Practices, Good Pharmacoepidemiology Practices, and the study was approved by the Ethics Committee of The First Hospital of China Medical University (Reference number [2018] 2018–153-2). Written informed consent was obtained from patients prior to study initiation. The study findings were reported in accordance with the guidelines for Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD).

Outcomes and Variables

For the primary objective, the outcome was the occurrence of AECOPD (ie occurrence versus no occurrence). For the secondary objective, the outcome was AECOPD severity (ie severe versus mild/moderate AECOPD). Potential candidate variables identified from the literature search that had >10% missing data were not considered (excluding variables such as drug use, comorbidities, and event-type variables, where missing data were interpreted, for example, as no drug use or no event). The full lists of candidate variables are shown in Tables S2 and S3.

Statistical Analyses

The full analysis set (FAS) comprised patients who fulfilled the inclusion/exclusion criteria with complete information for all candidate variables. Multivariable prediction models were used to construct the identification tools for AECOPD occurrence and AECOPD severity. For the development of the prediction model for AECOPD occurrence (primary outcome), all patients in the FAS (which comprised patients with stable COPD, AECOPD, or no COPD) were included in the analysis. For the development of the prediction model for AECOPD severity (secondary outcome), only patients with AECOPD were included. For each study outcome, three training models were built as the source of apparent performances. Performance measures used were the Akaike information criterion (AIC), Bayesian information criterion (BIC), adjusted R² and area under the receiver operating characteristic curve (AUC); a smaller AIC or BIC score, and a higher adjusted R² or AUC were desirable. A stepwise procedure was used to select candidate variables with significance levels <0.15 to enter the model, and <0.20 to stay in the model for multivariate analysis. Stepwise logistic regression is commonly used for variable selection¹⁶ and this method was prespecified during the study design. Age and sex were forced to stay in the model for the primary outcome; age, sex, and most recent exacerbation in the past 12 months were forced to stay in the model for the secondary outcome. The bootstrap method, which is recommended for optimism and overfitting correction,¹⁷ was subsequently used for internal model validation, and this procedure was repeated until sensitivity and specificity >0.70 was reached. Cut-off values, that is, the value that maximizes the Youden index (sensitivity + specificity – 1), were calculated to support the clinical utility of both AECOPD occurrence and AECOPD severity prediction models. Individual scores were derived using the logistic regression equations of the most optimal models. The study followed relevant statistical guidance.¹⁸ All statistical analyses were carried out on non-missing data using SAS version 9.4. Missing data were handled using a complete-case approach and data were not imputed unless otherwise specified. For categorical variables, unknown and missing data were presented as a separate category, with the denominator excluding unknown or missing values as appropriate.

Results

Patient Demographics and Baseline Characteristics

Of the 302 patients screened, two did not meet inclusion/exclusion criteria and one patient withdrew consent. A total of 299 patients were enrolled in the study, and 246 patients were included in the FAS (Figure 1). Baseline characteristics of patients in the FAS are presented in Table 1. Briefly, 74 (30.1%) patients had COPD, 100 (40.7%) patients had AECOPD, and 72 (29.3%) had no COPD. Among those who had AECOPD, 48 (48.0%) were mild/moderate while 52 (52.0%) were severe (Table S4). Mean (standard deviation [SD]) age was 64.1 (9.6) years and most patients were male (n=211, 85.8%); 126 (51.2%) had exacerbations in the past 12 months, and 12 (4.9%), 33 (13.4%), and 91 (37.0%) patients had exacerbations resulting in ≥1 emergency room treatment, hospitalization, and outpatient treatment, respectively.

Table 1 Patient Demographics and Baseline Characteristics (FAS)

Figure 1 Flow of participants through the study. Abbreviations: AECOPD, acute exacerbation of chronic obstructive pulmonary disease; COPD, chronic obstructive pulmonary disease; FAS, full analysis set.

Patient Core Symptoms

Among patients in the FAS, those with COPD or AECOPD had the highest severity scores (1.8–3.0) for breathing difficulty, shortness of breath, sputum evaluation and difficulty, and cough evaluation (Figure 2). COPD assessment test (CAT) symptoms with the highest severity scores (2.0–3.0) were breathlessness, phlegm, and being limited when performing household activities (Figure 3). For non-COPD patients, symptoms with highest severity scores (1.9–2.1) were related to cough and sputum (Figures 2 and 3). Symptoms with the greatest difference in severity scores between patients with AECOPD and patients with COPD were breathing difficulty (3.0 versus 1.8), chest congestion (2.2 versus 1.2), and chest tightness (2.0 versus 1.1) (Figures 2 and 3). Symptom severity scores were largely similar between patients with severe versus mild/moderate AECOPD (Figures 4 and 5) although being limited when performing household activities had a much higher CAT score in patients with severe versus mild/moderate AECOPD (3.2 versus 1.8) (Figure 5).

Figure 2 Severity scores for core symptoms (patients in FAS). We assigned the following values for phlegm color: 1=clear phlegm, 3=yellowish orange phlegm, 5=dark green phlegm. Abbreviations: AECOPD, acute exacerbation of chronic obstructive pulmonary disease; COPD, chronic obstructive pulmonary disease; FAS, full analysis set.

Figure 3 Severity scores for CAT symptoms (patients in FAS). Abbreviations: AECOPD, acute exacerbation of chronic obstructive pulmonary disease; CAT, COPD assessment test; COPD, chronic obstructive pulmonary disease; FAS, full analysis set.

Figure 4 Severity scores for core symptoms in AECOPD patients in FAS. We assigned the following values for phlegm color: 1=clear phlegm, 3=yellowish orange phlegm, 5=dark green phlegm. Abbreviations: AECOPD, acute exacerbation of chronic obstructive pulmonary disease; COPD, chronic obstructive pulmonary disease; FAS, full analysis set.

Figure 5 Severity scores for CAT symptoms present in AECOPD patients in FAS. Abbreviations: AECOPD, acute exacerbation of chronic obstructive pulmonary disease; CAT, COPD assessment test; COPD, chronic obstructive pulmonary disease; FAS, full analysis set.

Primary Outcome Analysis

Performance results of the training models and that after internal validation are summarized in Table 2 for the primary outcome. Model 2 was considered the most ideal model for identifying patients with AECOPD (sensitivity 0.84 [95% CI: 0.75–0.90], specificity 0.77 [95% CI: 0.71–0.85], AIC 245.82, BIC 277.37, adjusted R² 0.47, AUC 0.86) (Table 2), and had an intercept of –6.4549 (Table 3). The following variables were significantly associated with an increase in AECOPD occurrence (P<0.05) and entered the final model: GOLD classification, most recent exacerbation in the past 12 months, number of exacerbations that resulted in hospitalization in the past 12 months, phlegm color, respiratory rate, and chest tightness. Age and sex were not significantly associated with the occurrence of AECOPD (P>0.05) (Table 3). The optimal cut-off value corresponding to the Youden index was 0.374 (derived using the logistic regression equation of model 2), indicating that patients were categorized as test positive (ie potentially having AECOPD) when ≥0.374, or as test negative when <0.374 (Supplement 1).

Table 2 Model Performance Results for the Primary Outcome Analysis

Table 3 Occurrence of Exacerbation Multivariable Logistic Regression Model (FAS=246)

Secondary Outcome Analysis

Performance results of the training models and that after internal validation are summarized in Table 4 for the secondary outcome. Model 3 was considered the most ideal model for identifying severe AECOPD (sensitivity 0.90 [95% CI: 0.83–0.98], specificity 0.73 [95% CI: 0.60–0.86], AIC 136.93, BIC 157.77, adjusted R² 0.22, AUC 0.81) (Table 4), and had an intercept of –0.6810 (Table 5). The following variables were significantly associated with AECOPD severity (P<0.05) and entered the final model: exacerbation in the past 12 months, number of exacerbations that resulted in hospitalization, limited when performing household activities, and time of onset of symptoms. Age and sex were not significantly associated with AECOPD severity (P>0.05) (Table 5). The optimal cut-off value corresponding to the Youden index was 0.405 (derived using the logistic regression equation of model 3), indicating that patients were categorized as test positive (potentially having severe AECOPD) when ≥0.405, or as test negative when <0.405 (Supplement 2).

Table 4 Model Performance Results for the Secondary Outcome Analysis

Table 5 Severity of AECOPD Multivariate Logistic Regression Model (AECOPD Patients in FAS=100)

COPD Knowledge

Total COPD knowledge questionnaire (COPD-Q) scores (mean±SD) in the COPD (6.1±2.0) and AECOPD (6.3±2.1) groups were higher than that in the non-COPD group (4.2±2.6). The questions with the lowest proportion of correct answers were: “COPD medicines keep the disease from getting worse” (FAS: 13.0%; AECOPD patients: 12.0%), and “Stopping smoking will keep COPD from getting worse” (FAS: 9.8%; AECOPD patients: 10.0%). The questions with the highest proportion of correct answers were: “People with COPD may feel short of breath” (FAS: 78.0%; AECOPD patients: 90.0%), and “Cigarette smoking or second-hand smoke causes most COPD” (FAS: 71.1%; AECOPD patients: 79.0%) (Table S5).

Discussion

To the best of our knowledge, this is the first study comprising a broad spectrum of respiratory patients to develop a diagnostic model for identifying patients with AECOPD, using key patient characteristics and disease-related factors that were easy to obtain and applicable to daily clinical practice. The model demonstrated high clinical applicability, and may help clinicians (especially those in primary care) enhance their awareness and ability to diagnose patients with AECOPD, guide disease management, and reduce inappropriate prescribing and patient monitoring.

Many models have predicted future exacerbations, but none have detected current exacerbations. Herer et al reported prediction models for AECOPD using multidimensional indices spanning patient characteristics, symptoms, use of COPD maintenance medications, and number of exacerbations in the previous year, and reported sensitivity of 21.9–84.4, specificity of 51.6–87.1, and AUC of 0.50–0.78.¹⁹ Others have used a machine-learning algorithm to predict the risk of severe exacerbation, with internal validation yielding an AUC of 0.82.²⁰ In a systematic review of 27 predictive models from 25 studies, it was found that only two out of 25 studies validated their models, and only three out of 27 models used high-quality statistical approaches for model development and evaluation.¹³ In contrast, instead of predicting risk, our models were constructed to detect AECOPD occurrence and severity. Several previous studies have proposed models to detect AECOPD, using COPD-Q,²¹ patient-reported daily symptoms^14,22 or patient features;²³ however, all of these may be subjective and may limit model utility. For example, patient-reported symptoms require frequent monitoring and may result in poor patient compliance.¹⁴ The sensitivity (62.5–99.3%), specificity (36.0–100.0%), and AUC values (0.75–0.91) of these models varied widely.^21–23 In contrast, our models included data for both patient symptoms and disease-related characteristics, which can be easily collected by respiratory physicians, further enhancing their clinical utility. In our study, the optimal models selected for diagnosis of AECOPD and its severity demonstrated high sensitivity (0.84 and 0.90, respectively) and specificity (0.77 and 0.73, respectively), with high AUC values (0.86 and 0.81, respectively), and low AIC (245.82 and 277.37, respectively) and BIC (136.93 and 157.77, respectively) values, thus indicating strong diagnostic accuracy. Collectively, these features reflect high clinical applicability for our models.

In the primary model, variables that were significant risk factors for AECOPD occurrence were COPD disease-related characteristics, and signs and symptoms. COPD disease-related characteristics (GOLD classification, exacerbation in the past 12 months, and number of exacerbations resulting in hospitalization) were important discriminating factors in the detection of AECOPD. This is consistent with findings from the ECLIPSE study, where deterioration of lung function and a history of exacerbations were associated with the risk of subsequent exacerbations.^24,25 Other variables associated with AECOPD occurrence were chest tightness and respiratory rate, which are inherent pathophysiological manifestations of AECOPD.^1,26,27 Phlegm color was also associated with AECOPD occurrence; bacterial infections are common in AECOPD and are indicated by deepening sputum color.²⁸ Age is a common risk factor for AECOPD,²⁹ and previous studies have demonstrated that there is an increased risk of AECOPD in females.^24,30 As such, these variables were included in our model to increase sensitivity and specificity. However, age and gender, while showing correlation, were not significantly associated with occurrence of AECOPD in our model.

In the secondary model, the number of exacerbations leading to hospitalization was significantly associated with severity of AECOPD. This corroborates previous findings, which showed that a history of prior severe exacerbations resulting in hospitalization was positively associated with new severe exacerbations and mortality.^31,32 Interestingly, we found that exacerbation in the past 12 months was strongly associated with mild/moderate AECOPD, indicating that past exacerbation may be a protective factor. This may be due to a higher proportion of patients with mild/moderate compared with severe AECOPD in our study that sought outpatient treatment (79.2% versus 36.5%), although these values may be impacted by memory bias or the small sample size. History of outpatient treatment may have a positive impact on patient behavior and lead to standardization of medication, device use, and increased disease awareness, which in turn may improve disease management.³³ In addition, patients who are more engaged with COPD management may be more likely to seek professional healthcare advice when they perceive a worsening of their respiratory symptoms,⁹ as well as adhere to treatment,³⁴ which in turn is associated with lower rates of severe exacerbations.³⁵ Other important risk factors for severe AECOPD were being limited when performing household activities, and having morning symptoms. This is in line with previous studies, in which increased exacerbation symptoms (breathlessness, cough and sputum, and chest symptoms) were associated with reduced daily physical activity,³⁶ and the rate of severe exacerbation was higher in patients with severe morning symptoms.³⁷ Similar to the primary outcome model, age and gender were not associated with severe AECOPD.

The diagnostic tool for identifying patients with AECOPD may help clinicians, particularly those in primary care settings, to enhance their awareness of AECOPD risk factors and establish a diagnosis of AECOPD. The tool may be used as a supplement to established rating scales or existing predictive models, and may also improve patients’ self-management behavior. In this study, we found that all patients had poor COPD knowledge, especially pertaining to the progressive nature of COPD. Low levels of disease knowledge may impact how patients with COPD interpret the complex changes in their COPD symptoms,^9,38 and have been shown to be significantly associated with poorer self-management behavior.³⁹ Use of this diagnostic tool may help patients improve their knowledge of COPD and reporting behavior, thereby reducing future exacerbations, hospitalizations, and healthcare costs, and ultimately improve quality of life.⁹

This study has several limitations. Firstly, the model developed in this study was validated internally, but not externally. Future prospective studies need to be conducted to validate and improve the present diagnostic model. Nonetheless, the variables identified here were consistent with those associated with AECOPD in previous studies.^24,25,31,32 Secondly, potential candidate variables for model development were not selected based on structured criteria, which may have introduced bias. Nonetheless, these variables were identified by expert opinion and through a review of literature, as recommended by others.^40,41 Additionally, variables related to lung function and laboratory examinations, which may further improve the accuracy of the model, were not included in this study due to substantial missing data. These variables are not readily available in clinical practice, particularly in primary care settings, and are therefore of limited use in a diagnostic model. Finally, the small sample size in this study may have introduced bias, and this should be considered when applying these models in clinical practice.

Conclusion

By accessing readily available clinical patient and disease data, we developed diagnostic tools for identifying AECOPD and severe AECOPD. In addition, this study highlighted the need to improve patient education on COPD disease characteristics and AECOPD symptoms for more timely disease management. In primary healthcare settings, these diagnostic tools can aid physicians to detect AECOPD before further disease progression, and potential hospitalizations. Future studies should externally validate these models and directly compare their performance to other diagnostic tools.