The Development of a COPD Exacerbation Recognition Tool (CERT) to Help Patients Recognize When to Seek Medical Advice

Introduction

Exacerbations in chronic obstructive pulmonary disease (COPD) can increase the rate of loss of lung function,¹ and have a sustained effect on patients’ exercise capacity² and health status.³ They are associated with an increased risk of acute cardiovascular events such as heart attack and stroke,⁴ and an increased risk of death.⁵ There is evidence from across the world that many patients with COPD do not report, or under-report, exacerbations,^6,7 and that unreported exacerbations may have similar negative outcomes to those that are reported.^6,8–10 Unreported exacerbations are usually also untreated, and recovery may be slower and less complete than if they are treated.^6,8,11 There may be 2–5 times more unreported exacerbations in COPD than reported exacerbations.^6,7,9 Reasons for under-reporting these acute events are not known, but are likely to be multifactorial. In part, it may be because patients delay seeking medical attention in the hope that they will improve without active intervention. It may also be that the patient accepts exacerbations as a “normal” feature of their underlying COPD condition. Since exacerbations often follow upper respiratory tract infections, patients may also think that they are part of the same illness and the concept of a “chest cold” (ie, a cold that is accompanied by chest symptoms) is often familiar to patients, irrespective of language or geography. Common to all of these factors is a lack of understanding of the significance of early diagnosis and the need for treatment of exacerbations.

There is a need for simple-to-follow advice for patients, which would allow them to recognize when the worsening of their symptoms requires prompt medical assessment. A widely used clinical consensus definition states that an exacerbation is

A sustained worsening of the patient’s condition, from the stable state and beyond normal day-to-day variations, that is acute in onset and necessitates a change in regular medication in a patient with underlying COPD.¹²

Whilst conceptually useful, this definition lacks operational specificity, although very recently a time period of ≤14 days of symptoms has been suggested.¹³ A patient-reported diary card for COPD exacerbations (the EXAcerbations of COPD Tool [EXACT]) has been developed and validated,^14,15 but it contains 14 items, which may make it too large and complex for use as a simple guide for patients.

The objective of this study was to develop a valid COPD Exacerbation Recognition Tool (CERT) to provide patients with simple-to-follow guidance about when to seek medical attention when their symptoms worsen. The starting point for the development of this tool was the words and phrases that patients use to describe a COPD exacerbation and the endpoint was the identification of a focused set of symptoms that cover the range of symptoms that commonly worsen with an exacerbation.

Materials and Methods

A standard character set is used across China (simplified Chinese), but spoken languages and dialects differ, so the study ran in three sites located in the North, Southwest and Eastern parts of Mainland China. The study was conducted in accordance with the Declaration of Helsinki and ethics approval was granted for the three sites: The Second Affiliated Hospital of Shenyang Medical College (Nov 22, 2017), The Second Affiliated Hospital of Chongqing Medical University (no. 2018/07) and Hangzhou First People’s Hospital (no. 011-01). Patients from both urban and rural backgrounds were invited to participate. This was possible from a single site in each region, since mass migration from rural to urban environments allowed for identification of adequate numbers of patients from a rural background. The study had three stages that included item identification (Step 1), cognitive debriefing (Step 2), and quantitative analysis and item reduction (Step 3).

Step 1: Item Identification

Items were identified using standard qualitative methodology.¹⁴ Focus groups of 4–5 COPD patients, known to have experienced an exacerbation, were interviewed using a semi-structured approach. At least a third of those recruited were women. The interviews were conducted by a researcher trained in qualitative research. Concepts discussed covered the key domains of COPD exacerbations using items identified during development of the EXACT diary as seed items for the focus group discussions. The emphasis was on identifying noticeable change from the stable state. Areas covered included respiratory symptoms in terms of type, frequency and severity, and the impact of symptoms on sleep, daily activities, energy and emotion, perception and response to respiratory symptoms, awareness of change in their condition, response to these changes, cues used to seek care, and indicators of recovery. Three focus groups were performed in each centre.

Content analysis of the interviews used standard approaches.¹⁴ The transcripts were entered into qualitative research software (ATLAS.ti) and concept elicitation data were reviewed to identify key themes. Using a coding dictionary, words and phrases provided by subjects were highlighted and grouped into key themes and relationships. A saturation grid was created to track emergent information. Each new group of transcripts was assessed for the appearance of new concepts. Once no new concepts were documented, saturation was considered to be achieved. Experience of a similar study in COPD has shown that this may occur after ~20 individuals have been interviewed.¹⁶

The identified items from the three sites then were formed into a single draft item set, starting with the original words used by the patients. These words were then carefully condensed to make a single, clear description of how each item was described by patients.

Step 2: Cognitive Debriefing

A standard approach was used;^14,16 items that formed the draft item set were subject to cognitive debriefing through nine individual-person cognitive interviews (three per site) using new samples of COPD patients. Standard techniques were used; a “think aloud” procedure and “verbal probing” method were employed. Patients were given an interview guide after the interviewer had demonstrated how it was to be used and they were given the opportunity to practice until he/she felt comfortable with the process. The final form of the cognitive interviews was adjusted based on experience of the first interviews.

Interviews were audio-recorded for later transcription and analysis. They contained no patient identifiable information and were destroyed after the transcripts were made. The patients were asked to respond to all of the items, one at a time, then the interviewer reviewed the questions one-by-one. Qualitative data coding focused on the subject’s comprehension of the items, relevance of the content to their experience of exacerbations, ease of completion, and any suggestions regarding item language, phrasing and formatting. A panel of questionnaire experts and pulmonologists then reviewed all the items to identify duplicate, closely related or ambiguous constructs, and made a recommendation concerning the candidate items that went forward to the item reduction phase.

Step 3: Quantitative Analysis and Item Reduction

The candidate items were completed by patients who had not been in either a focus or cognitive debriefing group. The items were put into a questionnaire format. Each item had two types of response: 1. Severity during the exacerbation (Severity): “Not experienced”, “As usual”, “Mild”, “Moderate”, “Severe”; 2. Change from Usual State (Change): “Not experienced”, “As usual”, “Mild”, “Moderate”, “Severe”. Items were presented to any patients who could not read by a researcher (not a doctor) trained in non-directive questionnaire administration. A target of 150 patients was set to ensure an adequate ratio of subject to items for the Exploratory Factor Analysis (EFA). This is conventionally set at a minimum of 4:1. At the protocol design stage, the precise number of candidate items was unknown, but was unlikely to be much greater than 30, based on previous experience with developing the EXACT.¹⁴ Item reduction took into account frequency and magnitude of item responses and floor and ceiling effects. Tests of association were performed between item responses and demographic factors to ensure that items with a bias towards a particular group were excluded. EFA was used to identify domains. A panel consisting of statisticians, patient reported outcome specialists and clinicians, took into account all of the data to ensure an adequate balance of items to be taken forward for inclusion in the CERT. To ensure that the CERT was concise and easy for patients to use, a provisional target of 5 items was set a priori. Finally, tests of sensitivity and specificity were carried out to identify the minimum number of items that had to change for a patient to be advised to seek medical attention for their exacerbation. For the quantitative analysis, Statistical Analysis System (SAS) version 9.1.3 was employed.

Inclusion and Exclusion Criteria

Inclusion criteria were male or female patients aged ≥40 years with spirometrically confirmed COPD as per the Global Initiative for Chronic Obstructive Lung Disease (GOLD) 2016 criteria,¹⁷ with a treated COPD exacerbation within 3 months prior to the study visit and provision of informed consent.

Exclusion criteria were current diagnosis of asthma or clinically relevant bronchiectasis (patients with Asthma COPD Overlap were eligible), concurrent, uncontrolled or active medical condition in another organ, psychiatric condition, cognitive impairment, or any other reasons that would affect participation in the study or interfere with study evaluation in the investigator’s opinion.

Results

Steps 1 and 2

In Step 1, a total of 48 patients (25% female) participated. Their demographics are contained in Table 1. A total of 32 potential items were identified, falling under seven library headings: Cough and sputum (10 items and 43 quotes), Dyspnea and wheeze (7 items and 81 quotes), Exercise limitation (1 item and 6 quotes), Sleep (1 item and 25 quotes), Physical energy and emotion (5 items and 29 quotes), Diet (1 item and 19 quotes), Others (5 items and 48 quotes). These data have been previously presented in abstract form.¹⁸ After cognitive debriefing by a further 13 COPD patients and expert panel review in Step 2, 29 items were carried forward to Step 3.

Table 1 Patient Demographics

Step 3

In Step 3, 150 patients were recruited (equally split across the 3 centres; 21% female). The baseline demographics are presented in Table 1. The mean frequency response for the Severity score calculated for “moderate” and severe’ combined was 43%; range 2% (“sputum contains blood”) to 69% (“worsening cough”). The range of responses to the “severe” category was 0–17%, so this response category was not considered further. The mean frequency response for the Change score calculated for “moderate” and “severe” combined was 38%, with a range from <1% (“sputum contains blood”) to 61% (“worsening cough”). The range with the “severe” response category was 0–21%, so this response category was not considered further. The top 11 ranked items were common to both Severity and Change response options. Overall, the rankings were very similar between the two response options (Figure 1 and Supplementary Table 1.

Figure 1 Response rates (moderate and severe combined) for severity score and change score.

There was only a low degree of association between demographic factors and Severity or Change score. The highest shared variance with any item was 7.7% (between gender and severity of headache). In the 11 items that had the highest response rates, the average shared variance was gender 1.1%, age 1.1% and height 0.5%. Education level had a mean shared variance 4.5%, and a shared variance ≥5% was seen with five items using the Severity response option and seven items using the Change response option. To analyze this further, a comparison was made between the percentage of patients with low symptom scores in the lower and higher education groups. Low scores were seen in 44% of patients with education below high school, and in 46% in those with high school or college education.

The 11 items with the highest response rates were subjected to Exploratory Factor Analysis using varimax rotation. Two factors were identified. The item loadings were similar whether using Severity or Change scores (Table 2) but a little higher with the Change score than with the Severity score (Figure 2A and B) and their rank orders differed slightly between the two response formats (Figure 2A and B), but overall the two response options gave very similar factor solutions. Factor 1 contained six significant items and Factor 2 contained four. Only one item did not load onto one or other Factor (Item 5, “Hard to cough out sputum”). In general, the items in Factor 1 concerned activity and breathlessness, while in Factor 2 they concerned cough and sputum. The mean factor loadings and response rates using the two response options are summarized in Table 3. Taking into account the proportion of items in each factor, the factor loadings and the response rates, the expert panel identified 5 items to form the CERT: worsening cough, volume of sputum increased, shortness of breath, laborious breathing, and limitation of motion.

Table 2 Factor Loadings from an Exploratory Factor Analysis Using Moderate and Severe Responses to Severity of Item During an Exacerbation (Severity Score) or Moderate and Severe Responses to the Change in Item from the Normal State (Change Score)

Table 3 Mean Factor Loadings and Mean Response Rates for the 11 Highest Responded Items Combining Data from Both Response Formats (Severity During the Exacerbation and Severity of Change from the Usual State)

Figure 2 Item loading using responses to severity score and change score: (A) Factor 1, (B) Factor 2. *Indicates significant loading.

Tests of Sensitivity and Specificity of CERT Item Positive Responses

It was envisaged that the CERT would also include a recommendation about the number of symptoms that were moderately or significantly worse and should trigger physician assessment. A high number (such as 4 or 5) would risk low sensitivity and a low number (such as 1 out of 5) would risk a high number of false positives, so response rates of 2 out of 5 and 3 out of 5 needed to be tested for sensitivity and specificity. At this stage in CERT development and validation, a full study of CERT item responses using a new random sample of patients with exacerbations and stable patients was not possible, so a surrogate marker of an exacerbation was developed using data collected in Step 3. The patients were categorized into “exacerbating” or “not exacerbating” using their responses to the 10 items with the highest response rates. There was no previous work to guide the number of items that had to be positive for the patient to be categorized as “exacerbating”, although clinicians commonly make a diagnosis of an exacerbation based on the presence of two or more symptoms.

Two different surrogate exacerbation markers were created, one using any combination of ≥2 moderate or severe responses to the top 10 items, the other using any combination of ≥3 moderate or severe responses to the top 10 items. Sensitivity and specificity were then tested for the 4 different combinations (ie, two positive CERT item combinations and two different surrogate exacerbation definitions). The analyses are contained in Supplementary Tables 2–5. In summary, using the ≥2 CERT item positive model, the sensitivity for the two “exacerbator” surrogates was 91.8% and 98.4%, and specificity was 100% and 89.3%. Using a ≥1 CERT positive model, the sensitivity was 99.3% and 99.2%, but specificity was 62.5% and 36%. Based on this analysis, the recommendation is that the patients would be advised to seek medical attention if they experienced moderate or severe worsening of any combination of ≥2 of the five CERT items.

Expert Advisory Board

A GSK-sponsored advisory board of senior Chinese pulmonologists was held on January 152,021, to examine the contents of the proposed CERT (Supplementary Material). The advisors considered the data and the analyses, and agreed that the resulting content and response options of the CERT looked appropriate. There was also a view that high sensitivity was more important than high specificity. Although there was discussion of the choice of words and whether they would be suited for different regions of China, there was general agreement that whilst the precise words that patients use with their doctors may differ between regions, the simple descriptions in the CERT should be familiar to patients across the country. A second group of pulmonologists discussed the presentation and layout of the CERT and also addressed issues around the wording for use across all of China. As a result of these discussions, it was decided to include a simple pictogram for each item to aid patient understanding as illustrated in Supplementary Figure S1.

Discussion

This study was designed to create an evidence-based tool to provide patients with simple-to-follow guidance about when to seek medical attention when their COPD symptoms worsen. It identified five symptoms that were representative of the 29 symptoms that were reported by patients in the qualitative part of the study. We used two methods of scaling, one assessed the severity of the symptoms during an exacerbation, the other assessed the severity of change from the patient’s usual state. The factor loadings and responses rates were very similar with both response modes so “change from usual” was used in the wording of the CERT. The final selection of items was made by a panel of experts across the disciplines of questionnaire development, clinical content expertize and statistics. It drew on the results of the factor loadings and response rates shown in Table 3. An advisory board of senior pulmonologists critically reviewed the CERT and concluded that it had high face validity. A similar pattern of items can be seen in the subscales of the EXACT diary.¹⁵ The chosen items appear to be minimally influenced by demographic factors, particularly educational achievement, and age, so the CERT should be suitable for a wide range of individuals at risk of a COPD exacerbation. The advisory board had some concerns about whether the words that were used within the CERT would be understood across all regions of China, so for this reason the wording was simplified as much as possible, and pictograms were added.

The primary purpose of the CERT is to increase patients’ reporting of exacerbations, so good sensitivity is a requirement. Good specificity is also required to avoid triggering inappropriate medical attendances that would not only reduce patient and physician confidence in the CERT but also increase the burden of clinic or emergency room visits. At this stage in the development of the CERT, the resources were not available to perform a study to formally test its sensitivity and specificity, so we developed a surrogate method by categorizing patients’ responses to the items in Step 3 of the study as exacerbating/non-exacerbating. To mitigate the risk of bias, we carried out multiple tests using different numbers and combinations of items being worse, and compared those to different numbers and combinations of the 10 items most frequently reported as being moderately or severely worse with an exacerbation. These tests showed that worsening of two or more CERT items provided consistent estimates of sensitivity and specificity. This finding fits with clinical practice, since physicians typically enquire about 2 or more symptoms, as in: “Has your cough or breathlessness got worse?”. The time course of worsening was not addressed in this study, but a very recent consensus document suggested ≤14 days worsening¹³ and consideration should be made of including this in the instructions to the patients. It is important to recognize that the CERT is not a diagnostic instrument and does not set criteria for the presence of an exacerbation. It was designed to identify symptomatic markers that are suggestive of an exacerbation, so it is more analogous to a case-finding tool rather than a diagnostic tool. It is noteworthy that the experts in the advisory board felt that sensitivity was more important than specificity in clinical practice, and they also felt that the CERT may have a research application for case-finding in surveys of the prevalence of exacerbations, but that would require further study to determine the optimum number of items that should be positive.

The CERT was developed in Mainland China where Mandarin is used throughout the country, unified by a single common character set (Simplified Chinese), but there are a wide range of dialects and languages. We conducted the interviews in three different regions of the country and included participants who were not able to read or write or had only primary school education. The analysis and cognitive debriefing steps did not identify specific regional difficulties in meaning or understanding. Whilst there may be differences in the words that patients choose to express their symptoms, it is probable that the Simplified Chinese characters used in the CERT describe symptoms that patients can recognize, even though they may not use the specific words themselves. Although the CERT was developed in Chinese patients, this does not mean that it is a “Chinese questionnaire”, any more than instruments developed in United Kingdom, United States or Europe are “Western questionnaires”. Instruments such as the St. George’s Respiratory Questionnaire,¹⁹ the Clinical COPD Questionnaire,²⁰ and the COPD Assessment Test,²¹ have been translated from their language of origin into multiple others. There are standard approaches for translation, back-translation and linguistic validation to ensure that the meaning and intent of the items in a questionnaire are transferred reliably from one language to the next. The same process can be applied to the CERT. Whilst the wording of the CERT should be followed closely, the style, layout and choice of pictogram (if used) can be adapted to be appropriate for the country, language, and patient group in which it will be used. Figure S1 in the Online Supplement shows the CERT in Simplified Chinese and with an English translation superimposed on the layout developed for China, but other formats are acceptable. The objective should be to create a version that has a culturally appropriate style and is easy for patients to understand and use.

The intention of this study was to develop a tool to help patients in all GOLD groups recognize moderate and severe exacerbations. A potential limitation is that the patients included in Step 1 (Item identification) were mainly those who had severe exacerbations with hospitalization; few patients had moderate exacerbations. In part, this reflects current clinical practice in secondary and tertiary care hospitals in China where patients are often admitted if they present with an exacerbation. However, since the content reflects the content of the EXACT diary,^14,15 it is likely to have adequate sensitivity to detect the onset of moderate exacerbations, but further studies are needed to confirm that it can recognize them reliably.

Conclusions

The CERT has been developed to help patients recognize the onset of an exacerbation that may require treatment, but it should not be used in isolation. It should form part of an education programme in which COPD patients learn about life-style modifications including smoking cessation, diet and activity, the need for vaccination and adherence to their maintenance treatment. It will also need to be accompanied by information for clinicians so that they know what the CERT is, how it can help them in the management of their patients, and how can they best guide the patient to use the CERT.