External validation of the PUMA COPD diagnostic questionnaire in a general practice sample and the PLATINO study population

Introduction

COPD is a common condition that leads to physical disability and represents a major health problem worldwide.

Several studies have evaluated the prevalence of COPD underdiagnosis in different settings.^1–4 Lamprecht et al evaluated the underdiagnosis of COPD using different population-based studies and reported that approximately 81% of COPD cases remain undiagnosed and that there is considerable variation between the locations.² In contrast, Casas et al evaluated a high-risk primary care population from Latin America and found COPD underdiagnosis in 77% of cases when using a definition of post-bronchodilator FEV₁/FVC<0.70, and underdiagnosis in 73% of cases when using a lower limit of normal (LLN) for FEV₁/FVC definition.³ Underdiagnosis or non-diagnosis of COPD is important as these patients have significant limitations in their level of physical activity, reduced quality of life, increased mortality and also contribute considerably to the burden of health care.^5–8 Therefore, the approach of trying to identify the underdiagnosis of COPD is essential in order to reduce the burden of the disease, delay disease progression and improve health through smoking cessation support.

Primary care is an ideal setting for implementing search strategies to detect COPD cases. However, general practitioners (GPs) have to handle multiple diseases; therefore, the availability of a simple tool to identify COPD cases is extremely useful for these physicians.

Different risk factors and symptom-based prescreening questionnaires have been developed for the early detection of COPD. These questionnaires aid the selection of patients for spirometry screening to confirm definitively the diagnosis of COPD.

Even though there are validated COPD prescreening questionnaires^9–13 coming from developed countries they may not be applicable in the primary care or general population of developing countries such as those evaluated in the present study.

In an at-risk population visiting the primary care setting in four Latin American countries, Lopez et al assessed the accuracy of a prescreening score developed to detect COPD (the PUMA study).¹⁴ The simple score had an accuracy of 76% for detecting COPD (using a definition of post-bronchodilator FEV₁/FVC <0.70). To date, there has not been an external evaluation of this prescreening questionnaire.

The aim of this study was to validate externally the PUMA study prescreening COPD questionnaire in two different populations from Latin America and to compare its performance with the findings from the original PUMA study.

Methods

Development of the PUMA questionnaire in the original sample

The development of the seven-item COPD diagnostic PUMA questionnaire has been described in detail elsewhere.¹⁴ Briefly, PUMA was a multicenter, multinational, cross-sectional, non-interventional study conducted in the primary care setting of four Latin American countries: Argentina, Colombia, Venezuela, and Uruguay. At-risk patients completed the interview and performed acceptable post-bronchodilator spirometry. The simple PUMA prescreening questionnaire assessed risk factors and respiratory symptoms (gender, age, smoking pack-years, dyspnea, sputum, cough, and previous spirometry data) to select high-risk patients for spirometry screening.¹⁴ The score could range from 0 to 9 (higher scores indicating higher COPD risk) with 0–2 points assigned for each category. The points applied to each variable in the original PUMA questionnaire are described in detail elsewhere¹⁴ and a modified version is shown in Table S1. The best cut-point of the simple score according to Youden’s index to discriminate between persons with and without COPD was ≥5 points and the discriminatory power represented by the received operator curve (ROC) was 0.76.¹⁴

External validation samples

Patient data used for validation of the prescreening questionnaire were taken from the PLATINO population-based study¹⁵ and from the Hospital Maciel, Montevideo, primary care case-finding program; in both population samples, those patients fulfilling the original PUMA study inclusion criteria (≥40 years, current or ex-smokers [≥10 pack-years, ≥50 pipes/year or ≥50 cigars/year], and/or exposure to biomass smoke [wood or coal, for cooking or heating; exposure ≥100 hr/year]) were selected.¹⁴ In the Hospital Maciel sample, the criterion of exposure to biomass was not used because these data were not collected. For both the PLATINO population-based and the Hospital Maciel samples, the study designs were reviewed and approved, by the local Institutional Review Boards and Ethics committees, the studies were performed in accordance with the Declaration of Helsinki, and all patients provided written informed consent prior to participating.

Hospital Maciel, Montevideo, sample

The Hospital Maciel, Montevideo, implemented a case-finding COPD program from January 2010 to December 2015. The main objective of this program was to increase the availability of spirometry and enhance COPD diagnosis at the primary care level. The selection of patients for spirometry was based on GPs criteria for possible COPD. Each week, a respiratory technician from the pulmonary function laboratory of the Hospital Maciel attended the primary care centers to perform spirometry in those patients identified by the GPs. Patients completed a short risk factor and respiratory symptoms questionnaire that included questions on gender, smoking habits, respiratory symptoms, and prior spirometric testing. Spirometry tests were performed using a portable, ultrasound Easy One spirometer (ndd Medical Technologies Inc., Zurich, Switzerland) at baseline and 15 mins after inhalation of a bronchodilator (400 μg salbutamol), according to the American Thoracic Society criteria. A total of 1091 subjects completed the questionnaire and spirometry.

PLATINO study sample

The PLATINO study was a population-based survey performed in five Latin American cities; a total of 5314 subjects completed the interview and recorded spirometric measurements using a portable spirometer (Easy One spirometer, ndd Medical Technologies, Inc.) at baseline and 15 mins after inhalation of salbutamol 200 μg, according to the American Thoracic Society criteria of acceptability and reproducibility. Complete details of the methodology, participation rates and patients’ characteristics of the PLATINO study have been published elsewhere.¹⁶ Data were collected on several factors potentially associated with COPD, including gender, smoking habits, exposure to biomass smoke, respiratory symptoms, and prior spirometric testing.¹⁶ Copies of the questionnaires are available at the PLATINO website (http://www.platino-alat.org).

COPD definition

Definition and severity stratification of COPD proposed by GOLD was used: post-bronchodilator FEV₁/FVC <0.70. We also performed parallel analyses using the post-bronchodilator LLN for FEV₁/FVC as a criterion to define COPD.¹⁶

Statistical analysis

The characteristics of the patient populations in the two studies were described as absolute values and relative frequencies. The simple score was applied and then we calculated the area under the ROC and also the sensitivity, specificity, predictive positive value (PPV), predictive negative value (PNV), and the number needed to treat (NNT) for each point in the score. The Youden index (the sum of sensitivity and specificity-1) was used to determine the best cut-point. We calculated the power of our analyses, considering the obtained results, using the routine rocsize in Stata. For the study in Hospital Maciel, our sample had a power of 99.8% for the cut-off score of six points. In the PLATINO study, we had a power of 100% to estimate the parameters in any of the three major cut-off points (between a score of 3 and 5). All analyses were done using Stata 13.0 (StataCorp. 2013, Stata Statistical Software: Release 13, StataCorp LP, College Station, TX, USA).

Results

A total of 974 eligible individuals completed interviews and performed acceptable spirometry in the Hospital Maciel, Montevideo sample and 2512 subjects from the PLATINO baseline study (all sites) were eligible for the analysis. The flow chart for the patients is shown in Figure 1. Of these, 439/974 (45.1%) subjects in the Hospital Maciel sample and 450/2512 (18.7%) subjects from the PLATINO sample had COPD defined by post-bronchodilator FEV₁/FVC <0.70 (Figure 2). Using the LLN criteria, 374/974 (38.4%) subjects in the Hospital Maciel sample and 364/2512 (15.4%) participants in the PLATINO study had COPD (Figure 2).

Figure 1 The flow chart for the patients screening.

Figure 2 Prevalence of COPD according to different criteria (post-bronchodilator FEV1/FVC <0.70 and post-bronchodilator LLN for FEV1/FVC) using the Hospital Maciel, Montevideo, sample and the PLATINO study baseline population. Abbreviation: LLN, lower limit of normal.

Description of sample characteristics according to each population sample is shown in Table 1. In the Hospital Maciel sample, there was a higher proportion of women than men, nearly half of the subjects were ≥60 years of age, 64% were heavy smokers (>30 pack-years), about 75% of subjects reported respiratory symptoms (dyspnea, cough, and phlegm) and previous spirometry had only been performed in about 8% of subjects. In the PLATINO study, the proportion of men was slightly higher than women, around one-third of subjects were ≥60 years of age, subjects smoked less (about 60% smoked <20 pack-years), dyspnea was the most common symptom (~47%), only about 25% of subjects reported cough and phlegm, and previous spirometry had only been performed in 13% of subjects.

Table 1 Description of sample characteristics for the Hospital Maciel, Montevideo, and the PLATINO study samples

Description of COPD individuals’ characteristics from population sample according to the post-bronchodilator FEV₁/FVC <0.70 and the LLN definitions are shown in Tables 2 and 3, respectively. Using the post-bronchodilator FEV₁/FVC <0.70 definition, COPD patients in the Hospital Maciel sample were more likely to be men, approximately 60% were ≥60 years of age, heavy smokers (70% smoked >30 pack-years), with moderate to very severe airway obstruction (GOLD ≥2) and respiratory symptoms reported in >70% of patients (Table 2). Previous spirometry was recorded in only 11% of patients. In the PLATINO sample, COPD subjects were more likely to be men, ≥60 years of age (58%), there were fewer smokers (41% smoked <20 pack-years), with mild to moderate airway obstruction (GOLD 1–2) (Table 2). Dyspnea was the most common symptom (50%), while less than half reported cough and phlegm (~35%). Previous spirometry was recorded in only 22% of patients (Table 2). Similar results were found in both population samples for COPD defined using the post-bronchodilator LLN FEV₁/FVC definition (Table 3).

Table 2 Description of sample characteristics for the Hospital Maciel, Montevideo, and the PLATINO study samples in individuals with COPD defined using the post-bronchodilator FEV1/FVC <0.70 definition

Table 3 Description of sample characteristics for the Hospital Maciel, Montevideo, and the PLATINO study samples in individuals with COPD defined using the post-bronchodilator LLN FEV1/FVC definition

The area under the curve (AUCs) of the PUMA simple score with COPD as the outcome are shown in Figure 3A and B using the post-bronchodilator FEV₁/FVC <0.70 and post-bronchodilator LLN FEV₁/FVC definitions, respectively. Sensitivity, specificity, PPV, PNV, and NNT for each cut-point of the PUMA simple score are shown in Table 4. The mean accuracy (AUCs) of the PUMA simple score for detecting COPD using post-bronchodilator FEV₁/FVC <0.70 definition for the Hospital Maciel and PLATINO samples was 0.70 and 0.73 (Figure 3A), respectively. Using the LLN definition for the Hospital Maciel and PLATINO samples, the mean accuracy was 0.67 and 0.70, respectively (Figure 3B). The best cut-point according to Youden’s index (sensitivity + specificity −1) for the PUMA simple score was ≥6 in the Hospital Maciel sample and between 3 and 5 in the PLATINO sample, for both definitions. Using the post-bronchodilator FEV₁/FVC <0.70 definition, a subject having a score of <6 in the Hospital Maciel sample has a 70.9% chance of not having COPD (Table 4). In the PLATINO sample, a subject having a score of <5 has an 88.0% chance of not having COPD (Table 4). Similar results were observed with the LLN definition (Table 4).

Table 4 Sensitivity, specificity, PPV, PNV for each cut-point of proposed score (1 point for each category variable) using Hospital Maciel, Montevideo, and PLATINO study samples

Figure 3 Area under the ROC for score (1 point for each category variable) with COPD as outcome using: (A) post-bronchodilator FEV1/FVC <0.70 and (B) post-bronchodilator LLN for FEV1/FVC as definitions for the Hospital Maciel, Montevideo, the PLATINO study, and the original PUMA study baseline populations as samples.Abbreviations: LLN, lower limit of normal; ROC, received operator curve.

Discussion

In the present study, the validity of the recently developed PUMA COPD prescreening questionnaire was evaluated in two different patient samples: 974 at-risk patients from the Hospital Maciel primary care COPD case-finding program, and 2512 subjects from the PLATINO population-based study. The discriminative capacity of the PUMA questionnaire in both populations was similar to that found in the original study: using the COPD definition of post-bronchodilator FEV₁/FVC <0.70, the ROC_AUC was 0.70 and 0.73 for the Hospital Maciel and the PLATINO samples, respectively; using the LLN definition, the discriminative capacity of the questionnaire was lower at 0.67 and 0.70, respectively.

There are three externally validated prescreening questionnaires that assess risk factors and respiratory symptoms to select high-risk patients for spirometry screening: the COPD Diagnostic Questionnaire (CDQ),⁹ the Lung Function Questionnaire (LFQ),^10,11 and the COPD Population Screener (COPD-PS).¹² The predictive accuracy of these questionnaires in external validation studies was measured against post-bronchodilator FEV₁/FVC <0.70 and ranged from 57% to 72%.¹³

The CDQ is a symptom- and risk factor-based COPD prescreening questionnaire that was performed in a primary care-based US and UK.⁹ Possible scores range from 0 to 38 (higher scores associated with higher COPD risk). Two cut-points (16.5 and 19.5) have been proposed to select patients for spirometry screening based on ROC_AUC curves from the original development study.^9,17 The CDQ has been externally validated in five studies focused primarily on European and Australian primary care populations.^18–22 Most of the validation studies reported that a CDQ score of >16.5 had a sensitivity ranging from 80% to 91% and a specificity ranging between 24% and 49% for identifying those subjects who were subsequently found positive for COPD by spirometry testing.¹³ The use of a higher cut-point of 19.5 reduced sensitivity and PNV, but increased specificity and PPV. The discriminative ability of the CDQ to identify patients with and without COPD ranges from 0.65 to 0.72 (ROC_AUC).

The LFQ and the COPD-PS include risk factors and respiratory symptoms.^10–12 Scores range from 5 to 25, with lower scores associated with an increased risk of COPD. A threshold of ≤18 has been proposed as the cut-point for the risk of COPD that justifies using spirometry to confirm the diagnosis. This questionnaire was validated in a multicenter primary care study in 1288 smokers of at least 10 pack-years of exposure.²³ The study reported a high unacceptable spirometry rate (31%), as well as an estimated sensitivity of 88% and specificity of 25%.

The COPD-PS assigns scores to age, history of smoking, dyspnea, sputum production and functional limitations related to dyspnea. The score can range from 0 to 10.¹² A threshold of ≥5 has been proposed as a cut-point for the risk of COPD that justifies evaluation with spirometry. This questionnaire was validated in a Japanese population study (n=2357) and reported a sensitivity of 67% and specificity of 73% for a score ≥4, and 35% and 79%, respectively, for a score ≥5.²⁴ The discriminative ability (ROC_AUC) of the COPD-PS questionnaire to identify patients with and without COPD was 0.70 (score ≥4) and 0.57 (score ≥5).²⁴

The PUMA questionnaire assigns scores to gender, age, smoking history (pack-years), dyspnea, sputum, cough and data from prior spirometry. The best cut-point of ≥5 had a moderate discriminatory power ROC_AUC of 0.76 with a sensitivity and specificity of 74% and 65%, respectively.¹⁴ A subject with a score of <5 has a 91% (PNV) chance of not having COPD. The use of the PUMA score to identify patients with and without COPD has not been validated in other populations.

The present study evaluated the discriminatory capacity of the PUMA questionnaire score in individuals from two different settings (primary care and general population) using the original PUMA study inclusion criteria. The results from the Hospital Maciel, Montevideo (primary care sample), showed that the best cut-point of ≥6 had a moderate discriminatory power ROC_AUC of 0.70, a sensitivity of 69.9%, a specificity of 62.1%, a PNV of 70.9% and a NNT of 3 (one person diagnosed with COPD for every three persons assessed spirometrically). When the questionnaire was applied to the general population sample from the PLATINO study the discriminatory power was slightly better with a ROC_AUC of 0.73 and with three potential cut-points according to the Youden’s index: ≥3 (sensitivity 85.4%, specificity 46.9%), ≥4 (sensitivity 66.7%, specificity 66.5%), and ≥5 (sensitivity 51.5%, specificity 81.6%); the chances for not having COPD (PNV) at each cut-point were 93.3%, 89.9% and 88.0%, respectively. The NNT was 5 for scores ≥3 and ≥4, and it was 4 for a score ≥5.

This validation study, unlike the external validations of other questionnaires (CDQ, LFQ and COPD-PS)^18–24 found a discriminatory capacity that was similar to that reported in the original PUMA study, both in the primary care and in the general population samples. The results also show a performance of the PUMA score in this external validation in a similar range to the other prescreening COPD questionnaires (CDQ accuracy 0.65–0.73; LFQ accuracy 0.65; and COPD-PS accuracy 0.57–0.70).^18–24

This study has both strengths and limitations that need to be mentioned. The two populations used to validate the PUMA prescreening COPD questionnaire used the same inclusion criteria of the original study, with the exception of biomass in the Hospital Maciel sample. A particular strength of the study is that both the primary care and general population samples are large, and using two different population samples increases the spectrum of validity of the PUMA score. In addition to the lack of information on biomass in Hospital Maciel, other limitations are that the study validation samples included patients with and without respiratory symptoms, and this may underestimate the ability of the PUMA score to detect cases of COPD as defined by the GOLD document screening criteria (risk factors plus respiratory symptoms).¹⁶ This limitation also applies to the other prescreening COPD questionnaire studies that looked to detect spirometrically defined COPD in patients with risk factors but who were not necessarily symptomatic. Given the controversy of detecting asymptomatic airflow obstruction, this could be considered as an advantage of the prescreening COPD questionnaire in that asymptomatic patients would be less likely to score above the validated cut-points. Further research is needed to determine if the PUMA score performs better as a prescreening tool for COPD only in subjects meeting the GOLD screening criteria. In addition, both populations in the present validation come from Latin America, which limits the extrapolation of our results worldwide.

In summary, the results of this external validation study of the PUMA prescreening COPD questionnaire suggest moderate accuracy for detecting COPD using post-bronchodilator FEV₁/FVC <0.70 definition (discriminatory power 70–73%) in subjects ≥40 years of age with risk factors for COPD (smoking history and/or exposure to biomass smoke) from Latin America in a primary care or general population setting. Further research is needed on the performance of the PUMA score as a prescreening tool for COPD in other populations around the world.