External Validation and User Experiences of the ProPal-COPD Tool to Identify the Palliative Phase in COPD

Introduction

Despite severe morbidity and high mortality,^1,2 most patients with advanced chronic obstructive pulmonary disease (COPD) do not receive timely, adequate palliative care.^3–5 Palliative care aims to enhance quality of life of patients with a life-threatening disease through the assessment and treatment of physical, psychological, social and spiritual problems.⁶ It includes advance care planning (ACP), which enables individuals to define goals and preferences for current and future care.⁷

Identification of the palliative phase is a prerequisite to provide palliative care,^6,8 particularly in patients with organ failure like COPD.⁹ However, a major barrier is the unpredictable disease course in COPD, hampering accurate prediction of the timing of death.^3,10,11 Lung function parameters, such as forced expiratory volume in 1 second (FEV1) % of predicted do not correlate well with mortality of individual patients.¹² Also, survival models that have been developed to predict prognosis in stable patients, such as ADO, BODEx and DOSE, were not very accurate.^13,14

Palliative care guidelines recommend using the Surprise Question (SQ), a single-item tool: “Would I be surprised if this patient were to die in the next 12 months?”^8,15 It proved to be a useful tool to increase awareness among healthcare providers that patients are nearing the end of life.¹⁶ In hospitalized patients with COPD, however, its sensitivity was only 47%, indicating that many patients needing palliative care were missed.¹⁷ Furthermore, several tools including general as well as disease-specific indicators have been developed: the Gold Standards Framework Prognostic Indicator Guidance (GSF-PIG), the Supportive and Palliative Care Indicators Tool (SPICT) and the RADPAC indicators.^15,18,19 However, these tools are rather focused on the terminal phase and have not been validated for COPD.

To create a compact, practical guidance for healthcare providers to identify patients with COPD in need of proactive palliative care, the Propal-COPD tool was developed.²⁰ The tool combines the SQ with six clinical indicators. It was validated for patients admitted to the hospital for an acute exacerbation, as hospitalization increases mortality significantly and is a clear transition point in the disease course.^21,22 A high probability of death within one year was used as a proxy for palliative care needs, as they generally increase towards the end of life. Predicting 1-year mortality with high sensitivity (90%), the ProPal-COPD tool showed to be a promising tool to urge healthcare providers to initiate palliative care provision.

However, before implementing a prediction tool in clinical practice, it is essential to test the performance in another dataset than used for model development.²³ Additionally, it is relevant to examine the benefits of using the tool in practice and how the implementation can be optimized. Therefore, we aimed to externally validate the ProPal-COPD tool in a prospective cohort of hospitalized patients with COPD and to assess user experiences of healthcare providers.

Methods

Design

This validation study is part of the COMPASSION study, a cluster randomized trial in eight hospital regions across the Netherlands aiming to assess the effectiveness and implementation process of integrated palliative care.²⁴ Patients were recruited in the eight participating hospitals of the COMPASSION study and in two additional hospitals. Patients diagnosed with COPD and hospital admission for an acute exacerbation were invited to participate. Patients not able to complete questionnaires in Dutch, patients with severe cognitive decline (eg, dementia) and patients on the waiting list for lung transplantation were excluded. After receiving oral and written information about the study, written informed consent was obtained of all participants. The study was performed according to the Dutch law and Good Clinical Practice guidelines. The Research Ethics Committee of the Radboud University Medical Center Nijmegen concluded that this study does not fall within the remit of the Medical Research Involving Human Subjects Act (file number 2018–4390). The board of each participating hospital approved data collection. All data were stored and analyzed anonymized.

ProPal-COPD Tool

The ProPal-COPD tool consists of seven dichotomous indicators, of which each has a specific weight, together generating a total score (Table 1).²⁰ Two indicators are patient reported outcome measures: Medical Research Council (MRC) dyspnea score of 5 and Clinical COPD Questionnaire (CCQ) score higher than 3. Four indicators relate to clinical characteristics: FEV1 lower than 30% of predicted, presence of specific comorbidities, body mass index lower than 21 kg/m2 or weight loss and previous hospitalization for acute exacerbation. The last indicator is a negative answer to the SQ (“Would I be surprised if this patient were to die in the next 12 months?”). A score exceeding the cut-off value of −1.362 was previously published and corresponded in the development cohort with a high sensitivity (90%) and moderate specificity (73%) to predict death within 1 year. To create an online tool, we set the intercept to 0, resulting in a cut-off value of 2.539. During the first months of the study, the tool identified fewer patients than expected, and therefore after six months the cut-off value was lowered with one point to 1.5 (Table 2).

Table 1 Indicators of the ProPal-COPD Tool and Their Corresponding Weights

Table 2 Intercepts and Cut-Off Values of the Original, Converted and Adapted Model of the ProPal-COPD Tool

Data Collection

Participants filled in a questionnaire including demographic characteristics, smoking status, the MRC dyspnea score and CCQ score. A pulmonologist or COPD-nurse, involved in the care for the respective patient, gave their answer to the SQ and collected data on the four clinical ProPal-COPD indicators. Clinical baseline characteristics and all seven ProPal-COPD indicators were then entered by a healthcare provider in an electronic data management program (Castor edc). After twelve months of follow-up, survival status and, if applicable, date of death were collected from the medical records.

Experiences with the use of the ProPal-COPD tool were assessed using transcripts of monitoring meetings and semi-structured interviews with healthcare providers within the four intervention hospitals of the COMPASSION study.²⁴ They had been using the ProPal-COPD tool to identify patients who were offered palliative care conversations. To evaluate the implementation process, four monitoring meetings per region were held by EV and JB over the course of the study, and interviews were held by JB at study completion.²⁵ In these meetings and interviews, providers were asked to reflect on their experiences with the ProPal-COPD tool using open questions about the practical use, appropriateness of the patient selection, and the perceived effects of its use. Experiences with both the original and the adapted cut-off value were inquired. All participants provided written informed consent for participation in the study and anonymous use of their data.

Data Analyses

Participant characteristics were analyzed using descriptive statistics in SPSS version 25. Ordinary 2 × 2 tables were used to calculate sensitivity, specificity, positive and negative predictive values of the ProPal-COPD tool to predict 1-year all-cause mortality (calibration). These outcomes were calculated using the original cut-off value of the tool, the adapted, lower cut-off value and the SQ. We used a receiver operating characteristic (ROC) curve to calculate the area under the curve (AUC). An AUC (synonym for C-statistic) of 0.5 reflects no discriminative ability, and 1 reflects perfect discrimination. Differences in baseline characteristics between survivors and non-survivors were assessed using t-tests for continuous variables, Mann–Whitney U-tests for categorical variables and Chi-square tests for dichotomous variables.

Interview and monitoring meeting transcripts were deductively coded using user experience domains for eHealth interventions as proposed by Newton et al (2021).²⁶ They established working definitions for six domains. Acceptability refers to whether the intervention content, features, and delivery meet user expectations. Satisfaction refers to the user’s overall impression of the intervention and whether it meets their needs. Credibility refers to the extent to which the user perceives the intervention trustworthy and has the potential to work. Usability refers to the user’s perceived ease of use of the intervention based on technical factors. User-reported adherence refers to how and why the user did or did not follow the intervention or research protocol. Lastly, Perceived impact refers to the extent to which the user perceives the effect of the intervention’s impacts. Due to considerable overlap between Acceptability and Satisfaction, these domains were merged. Coding was done by one researcher (JB) and checked by a second researcher (YE). Disagreements were discussed until consensus was reached. Subsequently, a summary of each code was created and relevant quotations were selected. Findings were discussed within the research group until consensus was reached on the interpretation of findings.

Results

Participant Characteristics

Between May 2019 and August 2020, 825 patients admitted to the hospital due to an acute exacerbation of COPD were screened for eligibility. Eventually, 523 patients were included for analysis (Figure 1). Mean age was 70 years and 55.8% was female. Demographics and clinical characteristics of participants are presented in Table 3. Hundred patients (19.1%) died within 12 months after inclusion. Non-survivors were on average older, more often lived alone and more often received homecare, and had a lower lung function (FEV1% of predicted) than survivors.

Table 3 Demographics and Clinical Characteristics of Survivors and Non-Survivors and ProPal-COPD Tool Indicators

Figure 1 Flow diagram of study participants. AECOPD, acute exacerbation of chronic obstructive pulmonary disease.

Sensitivity and Specificity of the ProPal-COPD Tool

The ProPal-COPD tool had a low sensitivity (55.0%), and a medium to high specificity (73.3%) for predicting 1-year mortality. The positive predictive value was 32.7%, and the negative predictive value 87.3%. Using the lower cut-off value, sensitivity was higher (74.0%), but specificity lower (46.1%). The positive predictive value was 24.5%, and the negative predictive value was 88.2%. The ROC curve of the ProPal-COPD tool is presented in Figure 2. The AUC was 0.68 (95% confidence interval: 0.62–0.74).

Figure 2 Receiver operating characteristic (ROC) curve of the ProPal-COPD tool.

Sensitivity and Specificity of the Surprise Question

Sensitivity and specificity of the SQ alone were 56.0 and 73.0% respectively. The positive predictive value was 32.9%, and the negative predictive value was 87.5%. The AUC was 0.65 (95% confidence interval: 0.58–0.71).

User Experiences

Seven pulmonologists, nine COPD-nurses and one general practitioner shared their experiences about using the ProPal-COPD tool in interviews and monitoring meetings.

Acceptability/Satisfaction – Almost all participants liked having a tool to help them evaluate whether a patient entered the palliative phase.

In the hustle and bustle of the day, it’s nice if you get a reminder from time to time, so then I’m going to look differently. – COPD-nurse 5

Four participants found it particularly useful for less experienced healthcare providers. Two pulmonologists expressed that it helped them to make the decision more objectively, resulting in a shared understanding of prognosis. It was perceived essential by two pulmonologists to be able to screen not only hospitalized patients but outpatients as well.

Credibility – Before modification of the cut-off value, the tool was found to be too strict by healthcare providers across all four regions, resulting in missed patients in need of palliative care. After lowering the cut-off value, most of them indicated that the tool sometimes selected patients that were still too good for a palliative care conversation.

Sometimes I thought [the tool] underestimated and sometimes overestimated it. Because I’ve had patients with whom I’ve had advance care discussions and I’ve thought to myself, why am I having an advance care discussion here? And the next time, that patient comes hopping in. – Pulmonologist 2

Usability – Almost all pulmonologists and COPD-nurses considered the tool easy to use. Three of them preferred integrating the tool in the electronic medical record, obviating the need to open a separate webpage. According to the general practitioner, the tool was less suitable for use in primary care, because data are not available there for each indicator, eg, lung function.

The SQ was perceived as difficult to answer by three COPD-nurses. Furthermore, statements of two pulmonologists and two COPD-nurses suggested that the SQ was sometimes (wrongly) interpreted as a life-expectancy of less than one year.

What I did notice about the tool is that the surprise question is quite difficult for some, especially the nurses. To be able to estimate whether someone is still alive or not after a year. – Pulmonologist 1

User-reported adherence – After the study had ended, most participants indicated that with increasing experience they had stopped filling out the ProPal tool, but instead used some of its indicators, such as the SQ, to make their own assessment. They had gained more experience in recognizing palliative care needs, memorized the indicators, and realized that “it is not black and white”, partly because the tool did not perform as well as expected.

I do not think I would hold it so strictly to negative or positive, but rather that by looking at it that way, you judge a patient differently. – COPD-nurse 7

Healthcare providers of one region entered the ProPal-COPD indicators only for study purposes but did not use them for identification. They preferred to initiate palliative care if it became clear that treatment options such as pulmonary rehabilitation and bronchoscopic lung volume reduction were not possible anymore.

Perceived impact – Around half of the participants expressed that using the tool had made them look at their patients differently, as their awareness of the palliative phase had increased.

If you mark it huh, that palliative phase, that gives you some more insight that you can actually do something more instead of accepting that it’s just chronically bad with that patient. – COPD-nurse 2

Discussion

Main Findings

We externally validated and assessed user experiences of the ProPal-COPD tool to facilitate healthcare providers identifying the palliative phase in patients with COPD, hospitalized for an acute exacerbation. The ProPal-COPD tool showed to have mediocre predictive properties. Although healthcare providers considered the outcome of the tool not always correct, they generally did appreciate having such a tool, particularly for less experienced colleagues, because it increases awareness of the palliative phase and provides a shared understanding of prognosis.

Interpretation and Implications

There are several potential reasons why the ProPal-COPD tool did not confirm the promising data of the internal validation study and did not increase prediction of 1-year mortality compared to previously developed survival prediction models in COPD.¹⁴ First, prediction models always perform better in the derivation cohort than in a new population.²³ In the study of Duenk et al, the model was built with 11 indicators using data from 155 patients of which 30 died.²⁰ This relatively small number of ‘events’ might have led to overfitting of the model, limiting its performance in a new group. Second, the tool comprises dichotomous instead of continuous indicators, making the model less accurate as not all available information is used. For example, an MRC dyspnoea score of 4 or 5 reflects a small difference in clinical practice, but results in a big difference in the total score. Third, all deaths, regardless of cause, have been used. In previous research on the SQ and SPICT, leaving out acute and unexpected deaths led to increased sensitivity.²⁷ Fourth, our data collection took place during the COVID-19 pandemic. The pandemic may have caused changed mortality patterns due to COVID-19 infections and reduced transmission of common respiratory virus infections following public health measures, which may have influenced our results.^28,29

Despite the suboptimal performance, the systematic screening of patients using the ProPal-COPD tool was appreciated by healthcare providers as it made them more aware of palliative care needs. Examining the indicators in each patient, apart from calculating the score, proved to be beneficial in itself. Furthermore, the ProPal-COPD tool was found to be easy-to-use in the hospital, which could be further enhanced by integrating the tool into the electronic medical record.

The SQ had a similar low sensitivity and specificity as the ProPal-COPD tool using the original cutoff value. It is a simple tool, but was easily confused with life-expectancy, as was demonstrated by some interview statements. This confusion may be solved by use of the “Double Surprise Question”, adding a second question “Would I be surprised if this patient will be still alive after 12 months?” to the original SQ.^30,31

Although we used 1-year mortality to validate the ProPal-COPD tool, the primary use of the tool is to facilitate healthcare providers to proactively identify patients whose quality of life could be improved by a holistic palliative care approach. As palliative care needs in organ failure do not necessarily start one year before death and may fluctuate over time, it has been advocated not to pursue accurate mortality prediction but to use a needs based tool instead.³² For patients with heart failure, the I-HARP has recently been developed.³³ Finamore et al attempted to cluster patients with COPD by their symptoms, which could be a first step to development of such a tool specific for COPD.³⁴ However, due to limited time and financial resources, it is not attainable to provide a comprehensive person-centered assessment to all patients with COPD. Making a selection of patients most in need could help to distribute resources efficiently. Also, poor prognosis may define palliative care goals and topics to be discussed, in order to align care to the patient’s wishes. Further, it helps to overcome healthcare providers’ reluctance to talk about the end-of-life. Therefore, a tool that both identifies patients in need of palliative care and accurately predicts prognosis would activate healthcare providers to discuss end-of-life topics. Additionally, the shared understanding of prognosis may align goals and facilitate collaboration between healthcare providers in different care settings.

Strengths and Weaknesses of the Study

This multicenter and prospective study with a naturalistic and heterogenous population makes our findings generalizable to other COPD patient populations. With a relatively large sample size with 100 “events”, we met the minimum requirement for external validation studies, making our findings reliable.³⁵ Our study also has some limitations. First, following the development cohort of the ProPal-COPD tool, we only included hospitalized patients. This inhibits the generalizability of our findings to outpatients and primary care patients. Second, death rates were based on registration in medical records, since we had no access to official death certificates, and could have been incomplete. However, the risk of a missed deceased patient is very low because we assessed survival status of most patients in the medical records well beyond one year of follow-up and additionally we searched the internet for death advertisements (www.mensenlinq.nl). Third, we included patients from four intervention hospitals. The intervention could have theoretically influenced survival. However, in our effect evaluation, we did not observe any differences in survival between the intervention and control group.³⁶ Fourth, the SQ was answered by pulmonologists as well as by COPD-nurses. Interpersonal and interprofessional differences might have led to less precise prediction^16,37 but reflect normal clinical practice. Fifth, as we used existing qualitative data of the COMPASSION study for assessing user preferences, we may not have reached data saturation on all user experience domains.

Conclusion

The ProPal-COPD tool is easy-to-use and appreciated by healthcare providers, because screening with its indicators increases their awareness of the palliative phase and facilitates a shared understanding of the prognosis. However, the validity of the ProPal-COPD tool in predicting all-cause mortality within one year appears to be hardly superior than previously developed prediction models and the SQ. Future research should explore whether the predictive properties improve when using respiratory-related deaths or palliative care needs as outcome instead.