Performance and Clinical Utility of Various Chronic Obstructive Pulmonary Disease Case-Finding Tools

Introduction

Chronic obstructive pulmonary disease (COPD) is incurable but treatable. It is characterized by airway limitation and respiratory symptoms including dyspnea, chronic cough, and sputum production.¹ COPD is the third leading cause of death worldwide and imposes a substantial economic and social burden.² Furthermore, COPD remains largely underdiagnosed, and undiagnosed COPD is associated with unfavorable health outcomes.^3–5 Undiagnosed COPD is a considerable challenge for health-care systems, and patients with undiagnosed COPD have a heightened risk of early death.⁶ Increasing evidence supports the benefits of routine pharmacological treatment for the early stages of COPD; such benefits include reduced risk and severity of exacerbations, improved overall health, and slow disease progression.^7,8 Therefore, identifying patients with undiagnosed but clinically significant COPD is imperative.

Spirometry is regarded as the gold standard for COPD diagnosis; nevertheless, the US Preventive Services Task Force recommends against spirometry screening for COPD in asymptomatic adults.^9,10 Furthermore, despite evidence demonstrating that the underdiagnosis of COPD in primary care settings is a pertinent concern, spirometry is considered unsuitable for screening when resources are lacking owing to the high cost of equipment and the various requirements for appropriate technician training.¹¹ To overcome these limitations, several COPD case-finding tools have been developed and evaluated. The performance of such tools was summarized in a meta-analysis, which demonstrated that a symptom-based questionnaire, such as the COPD Population Screener (COPD-PS) questionnaire, ensured testing of symptomatic patients with COPD, but microspirometry was more accurate for disease diagnosis.¹² Prediction models are another COPD screening approach. Su et al developed and validated an accurate COPD prediction model, namely the P_COPD model, which uses a patient’s age, pack-years, percentage peak expiratory flow rate (PEFR), and COPD assessment test (CAT) score to accurately and rapidly identify at-risk patients with undiagnosed COPD who may require further diagnostic evaluation.¹³ Among these validated tools, microspirometry and the P_COPD model have been identified as feasible alternative tools for COPD screening in Taiwan.

Meta-analyses have summarized the performance of various COPD case-finding tools, including symptom-based questionnaires and microspirometry.^14,15 According to such analyses, several limitations remain regarding the heterogeneity in sensitivity and specificity related to patient smoking history, the inclusion of patients diagnosed with COPD, COPD prevalence, and the country in which the tools were used. For example, a study focusing on the COPD-PS questionnaire revealed that this questionnaire had the highest heterogeneity of all COPD case-finding tools, followed by spirometry.¹² Moreover, real-world evidence related to performance assessments of various COPD case-finding tools remains inadequate. Cognizant of these considerations, we conducted a retrospective validation study using a nationwide survey data set to evaluate the performance of 3 COPD case-finding tools, namely the COPD-PS questionnaire, the P_COPD model, and microspirometry, in a real-world setting to overcome the aforementioned limitations.

This study provides real-world evidence to identify optimal COPD case-finding practices in outpatient settings and ensures that physicians have improved access to up-to-date evidence when they encounter a symptomatic patient with suspected COPD.

Materials and Methods

Study Design and Procedure

This was a multicenter validation study involving retrospective data analysis. The study protocol and retrospective data analysis procedure for performance assessments of various COPD case-finding tools were approved by the Institutional Review Board of Changhua Christian Hospital (CCH IRB 210709). A flowchart of participant enrollment is presented in Figure 1. Participants were enrolled from 26 outpatient clinics from medical center, regional hospital, and district hospital in Taiwan, and all of the tests undertaken in these settings were completed on the same day. Participants’ demographic information, COPD-PS questionnaire scores, CAT questionnaire scores, microspirometer measurement data, and diagnostic spirometer measurement data were obtained from the relevant clinics. COPD was diagnosed and COPD severity was classified in accordance with GOLD definitions. The reference criterion for COPD was a post-BD FEV1/FVC ratio of <70%, and COPD severity was classified according to the predicted post-BD FEV1 percentage. Data related to all the variables were collected by and stored on an integrated automatic evaluation system, namely the SMART system (Self-Motivated Awareness Respiratory Tele-healthcare; Manifold Health Tech, Zhubei, Taiwan). The collected variables were demographic data; symptoms; FEV1, FVC, and FEV1/FVC ratio (confirmative conventional spirometry); FEV1, FVC, PEFR, and FEV1/FVC ratio (microspirometry); COPD-PS score; and CAT score. The scoring methods of the case-finding tools as listed below:

• COPD-PS questionnaire is a five-item questionnaire that was validated for screening individuals who are at high risk of COPD. It is composed of age, smoking history and three COPD-related symptoms (breathlessness, productive cough, and activity limitation). The score ranges from 0 to 10, a cutoff point of 4 has been found to be useful for COPD screening in previous study.¹⁶
• The CAT is an eight-item questionnaire that measures the impact of COPD on the patient. It is composed of COPD-related symptoms, including cough, expectoration, dyspnea, chest tightness, confidence in leaving home, limitation of daily activities, the quality of sleep and levels of energy. The score ranges from 0 to 40, with 40 being the most severe impact on health status.¹⁷
• The model and the manufacturer of the microspirometer, Spirobank Smart, has been reported in our previous study. In brief, the Spirobank Smart device (Spirobank Smart, MIR, Rome, Italy) is able to connect to smartphone apps through Bluetooth for the seamless recording of lung function parameters. In addition, Spirobank Smart is performed by trained, qualified personnel, including nurse and physician. The best FEV1 and FVC values of 3 attempts to meet ATS/ERS acceptability and repeatability standard.¹⁸
• P_COPD, a model to identify undiagnosed at-risk patients with COPD. Age, smoking pack-years, CAT score, and percent predicted PEFR) were used for establishing the prediction model. Su et al use the P_COPD for COPD screening among people ages 40 and older with respiratory symptoms and smoking history (≥20 pack-years). And P_COPD ≥ 0.65 were recommend for identifying subjects at risk of COPD in previous study.¹³
• The reference criterion for COPD was defined as a postbronchodilator (post-BD) forced expiratory volume in 1s/forced vital capacity less than 0.7 (FEV1/FVC< 0.70) determined by confirmative conventional spirometry.¹

Figure 1 Study Procedure.

Study Participants

This study was conducted by the Taiwan Society of Pulmonary and Critical Care Medicine. This society, commissioned by the Taiwan Ministry of Health and Welfare, recruited the participants and conducted the study. This retrospective study used data from Jan 2019 to December 2019. The patient selection criteria and clinical data were presented in a previous article.¹⁶ In brief, the inclusion criteria for participants were as follows: being aged ≥40 years, reporting ≥10 pack-years, presenting with chronic respiratory symptoms (cough, phlegm, or dyspnea, or a combination thereof), and not having a COPD diagnosis. Individuals who did not undergo post-BD spirometry and were unable to correctly operate the microspirometer were excluded. A total of 385 participants completed all of the tests, and their data were analyzed.

Statistical Analysis

Data are presented as percentage and mean ± standard deviation for categorical and continuous variables, respectively. Student’s t test and the chi-square test were used to compare the differences between the COPD and non-COPD groups. We also calculated the area under the receiver operating characteristic (ROC) curve for the COPD-PS questionnaire, the P_COPD model, and a spirometer for diagnosing COPD, with the FEV1/FVC ratio obtained through confirmatory conventional spirometry as the standard. Youden’s index was a statistic which is used to assess the best compromise between the sensitivity and specificity of a test. It was calculated to define the best cut-off point of COPD-PS questionnaire score, P_COPD prediction model score, and FEV1/FVC ratio determined by Spirobank Smart for COPD detection in our study population. The sensitivity, specificity, positive predictive values (PPVs), negative predictive values (NPVs), and ROC curve values were subsequently used to evaluate the performance of the screening tools in distinguishing between individuals with and without COPD. Furthermore, we assessed the clinical application of various COPD case-finding tools by using a decision curve analysis and clinical impact curve. Both univariate and multivariate logistic regression analysis were performed to determine the association between potential risk factors and COPD prevalence. In addition, the multivariate logistic regression using the variables found to be statistically significant in univariate analysis with backward stepwise elimination. All statistical analyses were performed using SPSS 22 software (IBM, Armonk, NY, USA) and R software (v.i386 3.6.2; https://www.r-project.org). A two-tailed P value of <0.05 was considered to denote statistical significance.

Results

Patient Characteristics

The demographic characteristics of the enrolled participants (101 with COPD categorized as the COPD group; 285 without COPD categorized as the non-COPD group) are presented in Table 1. A male preponderance (94.5%) was observed in our study. Compared with the non-COPD group, the COPD group was older, had a lower body mass index, and reported more pack-years. The COPD group had a significantly higher CAT score (12 ± 1), COPD-PS questionnaire score (6.2 ± 0.2), and P_COPD model score (0.75 ± 0.02) than did the non-COPD group (CAT score: 9 ± 0; COPD-PS questionnaire score: 4.8 ± 0.1; P_COPD model score: 0.49 ± 0.03; all P < 0.001). The pre-BD FEV1/FVC ratio determined using spirometry was significantly lower in the COPD group (63.16 ± 1.31) than in the non-COPD group (81.34 ± 0.44; P < 0.001). The COPD group had a lower post-BD FEV1/FVC ratio (58.88 ± 0.90) than the non-COPD group had (post-BD FEV1/FVC ratio: 81.00 ± 0.42). The demographic variables differed significantly between the groups.

Table 1 Participant Demographics and Characteristics

ROC Curves and Diagnostic Accuracy of Various COPD Case-Finding Tools

The areas under the ROCs and 95% confidence intervals (CIs) of the 3 aforementioned COPD case-finding tools in distinguishing between individuals with and without COPD were as follows: COPD-PS questionnaire (0.908, 95% CI = 0.87–0.95), P_COPD model (0.788, 95% CI = 0.74–0.84), and pre-BD FEV1/FVC ratio determined using microspirometry (0.726, 95% CI = 0.67–0.78; Figure 2).

Figure 2 Receiver-operating characteristic (ROC) curves for Spirobank Smart, and PCOPD, COPD-PS. (A) Spirobank Smart (FEV1/FVC); (B) PCOPD; (C) COPD-PS.

Clinical Application of Various COPD Case-Finding Tools

The related decision curves are illustrated in Figure 3A. The net benefit of microspirometry was greater than that of the COPD-PS questionnaire and P_COPD model in terms of threshold probability. Microspirometry had marginally better prediction performance than the other tools had. The clinical impact curves for the 3 COPD case-finding tools are illustrated in Figure 3B–D. These curves reveal the estimated numbers of participants deemed to be at a high risk of adverse outcomes, and the true positive value was in the range of 0.0 to 1.0. For example, at a 20% risk threshold, out of 1000 patients screened, approximately 400 would be deemed high risk through microspirometry analysis and >400 would be deemed high risk using the P_COPD model and COPD-PS questionnaire. Approximately 300 of these true COPD cases were identified using all 3 of the models. The results suggested that microspirometry is superior to the other tools.

Figure 3 Decision curve analysis and clinical impact curves for Spirobank Smart, and PCOPD prediction model, COPD-PS. (A) Decision curve for Spirobank Smart, PCOPD, and COPD-PS; (B) Clinical impact curves for Spirobank Smart; (C) Clinical impact curves for PCOPD; (D) Clinical impact curves for COPD-PS.

Threshold Values and Corresponding Predictive Performance of Various COPD Case-Finding Tools

The corresponding predictive performance levels of various COPD case-finding tools are presented in Table 2. According to Youden’s index related to the ROC analysis, a spirometry-derived FEV1/FVC ratio of <74% for a patient at high risk for COPD had the best predictive performance [area under the curve (AUC): 0.871 (0.78–0.89), sensitivity: 84.20%, specificity: 90.14%, PPV: 75.22%, and NPV: 94.12%]. Although a P_COPD score of ≥0.74 had a high specificity, its sensitivity was lower than that of microspirometry and the COPD-PS questionnaire, A COPD-PS score of ≥5 had a higher AUC value (0.64) than had a COPD-PS score of ≥4 (0.599) but had low sensitivity and specificity for identifying COPD. Compared with the other COPD case-finding measures, a microspirometry-derived FEV1/FVC ratio of <74% achieved the best balance between sensitivity and specificity and also had the highest predictive ability.

Table 2 Sensitivity, Specificity, PPV, NPV, and Area Under the Receiver Operating Curve for Spirometry, the PCOPD Model, and the COPD-PS Questionnaire

Associations of FEV1/FVC Ratio Determined Using Various Case-Finding Tools and Participant Variables with COPD Prevalence

Age, reporting ≥50 pack-years, and CAT category were positively associated with COPD prevalence; relevant details are listed in Table 3. After multivariate adjustments, we observed that a microspirometry-derived FEV1/FVC ratio of <74% (odds ratio [OR] = 43.88; 95% CI = 21.34–90.21), P_COPD score of ≥0.74 (OR = 4.91; 95% CI = 2.81–8.55), and COPD-PS score of ≥5 (OR = 2.76; 95% CI = 1.45–5.27) remained significantly associated with COPD prevalence.

Table 3 Univariate and Multivariate Logistic Regression for the Associations of FEV1/FVC Ratio with COPD Prevalence

Discussion

The principal findings of this study are as follows: First, in terms of performance and clinical utility, microspirometry was better than the P_COPD model and COPD-PS questionnaire. Second, the optimal FEV1/FEVC ratio threshold determined through microspirometry was 74%, that of the P_COPD model was 0.735, and that of the COPD-PS questionnaire was 5. Third, patients with a microspirometry-determined FEV1/FVC ratio of <74% (OR = 43.88; 95% CI = 21.34–90.21), P_COPD score of ≥0.74 (OR = 4.91; 95% CI = 2.81–8.55), and COPD-PS score of ≥5 (OR = 2.76; 95% CI = 1.45–5.27) exhibited a high risk of COPD.

Our results revealed that microspirometry was more accurate at diagnosing COPD than the other tools examined. This finding is similar to findings in relevant meta-analyses; however, various sensitivity, specificity, and AUC values as well as microspirometry thresholds have been reported elsewhere.¹² The performance measurements, including AUC, revealed that microspirometry had higher diagnostic accuracy in studies that recruited symptomatic smokers aged ≥40 years than in studies that recruited asymptomatic participants. Therefore, microspirometry is suitable for symptomatic patients with a smoking history and should not be used on asymptomatic participants.^12,19–21 Although, traditional spirometry is the gold standard for diagnosing COPD, it remains underused or unavailable in primary care settings or non-specialized areas due to time-consuming procedures, labor-intensive, and requires well-trained professionals for its execution. Microspirometer is a light weight and cheap device, that has the advantages of user friendly, time-saving features, the requirement of less patient effort, and which render them useful for COPD early detection. However, the aforementioned studies differed from the present study in that they used FEV1/FEV6 ratio instead of FEV1/FVC ratio. Consequently, a further comparative study that examines the performance differences between FEV1/FEV6 ratio and FEV1/FVC ratio for determining COPD using microspirometry in symptomatic smokers is required. Studies have reported that lower education levels in women, older age, and a prior diagnosis of asthma are associated with an increased risk of COPD among never smokers, suggesting that symptomatic never smokers should be included in clinical surveillance and screening efforts related to COPD.²² Evidence on the feasibility of microspirometry screening for COPD among symptomatic never smokers remains unclear; further investigation is thus required. Compared with conventional spirometry, microspirometry is a less expensive and more convenient tool for diagnosing COPD in symptomatic smokers.

The 5-item symptom-based COPD-PS questionnaire is used to identify individuals that likely have COPD.²³ This is the first study conducted in Taiwan to validate the use of this questionnaire for the early detection of COPD among symptomatic participants with a smoking history. The AUC of the COPD-PS questionnaire with a threshold of ≥4 was lower for our Taiwanese population (0.603) than the AUCs for Japanese and Spanish populations (Japan: AUC = 0.70; Spain: AUC = 0.65–0.79).^16,24,25 The AUC of the COPD-PS questionnaire with a threshold of ≥5 was lower for our Taiwanese population (0.65) than it was for a Greek population (AUC = 0.79), but it was the same as that for a US population (AUC = 0.65).^26,27 The reason for such differences is unclear, but participant backgrounds, study settings, and differences between countries may have affected the thresholds and performance of the COPD-PS questionnaire. In our study, despite the low specificity and PPV at thresholds of 4 or 5, the NPV was sufficiently high; this finding indicates that the COPD-PS questionnaire with a threshold of 4 or 5 is a useful screening tool. Moreover, a simple and self-scored tool, such as the COPD-PS questionnaire, for patient screening may lead to increased awareness, earlier symptom recognition, and the use of conventional spirometry for accurate diagnosis. Therefore, the COPD-PS questionnaire is useful for COPD screening because it can more expediently identify symptomatic patients compared with microspirometry; this expedience is crucial in resource-constrained settings and for routine patient monitoring.

The combination of a peak expiratory flow (PEF) device and a questionnaire has been investigated in relevant studies. Several test combinations have been proposed in COPD diagnostic research, including the P_COPD model, the CAPTURE model, and a new point-system analysis that incorporates handheld flow meter measurement with an international primary care airway group questionnaire.^11,13,28 In these studies, the combination of a questionnaire and PEF device achieved diagnostic performance superior to that of a PEF device alone. Studies have indicated an optimal threshold of 0.65 for the P_COPD model, with high specificity indicated (90%). However, our finding demonstrated that a threshold of 0.65 resulted in a lower specificity (65.85%) and a smaller area under the ROC curve (0.710) compared with a threshold of 0.74. This difference may have been caused by variations in inclusion criteria. Moreover, the present study was a nationwide study, whereas patients from a single medical center and a single specialist site were included in the aforementioned studies. Furthermore, standard instructions regarding how to use a peak flow meter and other devices may influence the measurement and diagnostic accuracy of PEFR in various clinical settings. Although COPD case-finding with a questionnaire and a PEF device has been deemed practical despite the aforementioned parameter differences, a microspirometry-derived FEV1/FVC ratio of <74% outperformed the P_COPD in our study. COPD case-finding through a combination of a questionnaire and microspirometry has been suggested to be suitable for use in health-care settings. Studies have demonstrated that a dual-combination assessment tool, namely VitalQPlus, which incorporates the COPD-PS questionnaire and the COPD-6 screener with an FEV1/FEV6 ratio of <75%, could be used by physicians to identify individuals at risk of COPD and to select patients for conventional spirometry.²⁹ This observation is similar to that in a meta-analysis; the authors of that meta-analysis suggested using a questionnaire as a prescreening test for microspirometry to improve COPD screening accuracy.¹² However, the prescreen use of a COPD-PS questionnaire followed by microspirometry with FEV1/FVC ratio calculations has not been validated and requires further investigation.

The GOLD criteria may underestimate and overestimate disease prevalence in young and older adults, respectively.^30,31 Although the lower limit of normal (LLN) measurement has been suggested to aid in COPD diagnosis, LLN-based criteria for diagnosing COPD may lead to COPD underdiagnosis in symptomatic patients.⁴ Furthermore, the LLN-based criteria generated more false negatives than did the conventional GOLD criteria; these false negatives led to the undertreatment of patients with COPD during different disease stages (eg, GOLD I and II). Additionally, LLN- based criteria tend to categorize elderly individuals with mild obstruction as not having COPD.³² In our study design, we used a case-finding strategy based on symptom screening. The conventional GOLD definition may therefore be the appropriate choice for reducing COPD underdiagnosis according to our screening strategy. Therefore, we suggest that instead of the conventional GOLD measure, an FEV1/FVC ratio of <0.74 be used along with microspirometry for optimal threshold determination. This optimal threshold resulted in an ideal balance between false positive and false negative results for our study population, thereby reducing the possibility of misdiagnosis.

This study has some limitations. First, retrospective data were collected from a limited number of participants receiving treatment in hospital-based facilities; thus, such individuals may not reflect the entire COPD population. Further evaluation of these alternative COPD case-finding tools is required in primary care settings to expand the tools’ generalizability. Second, the number of participants from primary care settings was smaller than that from hospital settings; this discrepancy might have limited the generalizability of our findings to various outpatient settings. Third, a considerable proportion of the nonsmoking population may exhibit COPD symptoms; consequently, future studies evaluating environmental risk exposure, such as from secondhand smoke or ambient particulate matter, may be of interest.

We evaluated the performance and clinical utility of 3 COPD case-finding tools in real-world settings. Microspirometry is the most accurate alternative to conventional spirometry. The P_COPD model and COPD-PS questionnaire are useful for identifying symptomatic patients likely to have COPD. The COPD-PS questionnaire is recommended as a pretest for microspirometry. High specificity may be required when selecting the most suitable alternative COPD case-finding tool. Various COPD case-finding tools can offer substantial assistance in resource-limited outpatient settings and enable the earlier detection of symptomatic patients with COPD.