Development of a Standard Set of Outcome Domains and Proposed Measures for Chronic Obstructive Pulmonary Disease in Primary Care Physical Therapy Practice in the Netherlands: a Modified RAND/UCLA Appropriateness Method

Introduction

Chronic obstructive pulmonary disease (COPD) is one of the common chronic health conditions of patients visiting a physical therapist in primary care practice in the Netherlands.¹ In 2017, a total of 35,227 patients with COPD were treated by primary care physical therapists.¹ Research showed that pulmonary rehabilitation can be beneficial for patients with COPD, for improving domains in health status including muscle function, cardiovascular function and reducing exacerbations.^2–4 Physical therapy is a key component of pulmonary rehabilitation for the treatment of patients with COPD.^5–7 These outcome domains should be part of clinical practice. Standardization of outcome domains and proposed measures in a standard set opens the opportunity to learn from differences in treatment outcomes which can be used for improving quality of care.

Measures for outcomes measurement and quality improvement can focus on the patient’s health status, e.g. scored with patient-reported outcome measures (PROMs), patient-reported experience measures (PREMs) and/or physical performance measures.^8–10 Interpretation of these measures over time can be used in the interaction between a patient and physical therapist (e.g. shared decision-making, goal setting and monitoring). Furthermore, physical therapists can learn from routine data collection to evaluate treatment effects and to compare differences in treatment effects between peers or other practices. Finally, a standard set of outcome domains and measures provides a basis for the development of quality indicators. Campbell et al (2003) defined a quality indicator as: “retrospectively measurable elements of practice performance for which there is evidence or consensus that it can be used to assess quality of care proved and hence change it”.¹¹ Quality indicators consist of structure (e.g. availability of a pulse oximetry device in the exercise area), process (e.g. guideline adherence, shared decision-making and goal setting) and outcome measures (e.g. quality of life or number of exacerbations).¹¹

Successful implementation of a standard set of quality indicators in daily practice is challenging.¹² The first step in the development of quality indicators is the selection of a standard set of measures, and stakeholder engagement in reaching consensus is a key component for quality improvement initiatives.^13,14 This stresses the value of including all stakeholders in the development process.

The population of patients with COPD is heterogeneous with variation in the number of comorbidities, different levels of functional impairments, and as a result differences in quality of life over time.^15,16 There is an increasing interest in stratification of heterogeneous patient groups to identify clustered characteristics within this group.^17–19 By identifying subgroups for patients with COPD, the outcomes of physical therapy treatment can be predicted more accurately. Identifying subgroups is therefore useful to validate the comparability between treatment outcomes in physical therapy practice. Furthermore, stratified care enables the opportunity to provide personalized care by choosing relevant interventions for patients with COPD based on their characteristics.

The aim of this study was to develop a standard set of outcome domains and proposed measures for patients with COPD in Dutch primary care physical therapy practice. The final set of proposed measures should be accepted as having expected added value in clinical practice. The measures should enable comparing differences in treatment outcomes at the level of the individual patient (monitoring and clinical decision-making) and learning from differences in aggregated treatment outcomes of individual physical therapists or groups of physical therapists, for quality improvement purposes, and for public reporting. With including a proposal to stratify patients in subgroups, we aim to guarantee personal care and to enable comparing treatment outcomes between matched patients based on their characteristics.

Methods

Design and Setting

A RAND UCLA modified appropriateness method was conducted in primary care physical therapy practice in the Netherlands.²⁰ A mixed method approach was used between October 2016 and July 2017. Eight separate steps were performed to select eligible measures (Table 1). The selection procedure was guided from the perspective of the quality of care. This means that we focused on the content of care and the goals related to the treatment outcomes but also on the structure and process of care and patient experiences. Concerning the selection procedure for the outcome domains, we first focused on the Dutch physical therapy guideline for COPD.⁵ The guideline describes clinically relevant outcome domains using the International Classification of Functioning (ICF).^21,22 The ICF model is commonly used in physical therapy to determine outcome domains for patients based on their diagnosis, including functions (e.g. decreased exercise tolerance), activities (e.g. physical activity) and participation (e.g. quality of life).²² A priori, our main focus was on the ICF outcome domains to select a standard set of proposed outcome measurements including patient-reported outcome measures (PROMs) and physical performance measures. However, because we hypothesized that structure and process measures, including patient-reported experience measures (PREMs), can also have an added value for evaluating quality of care, these were also included in the literature search. In order to stimulate stakeholder engagement, an advisory board was formed with one representative of patients, i.e. the Lung Foundation Netherlands (Longfonds), representatives of physiotherapists: the Royal Dutch Society for Physical Therapy (KNGF), the Association for Quality in Physical Therapy (SKF) and two representatives of Dutch health insurance companies: CZ Health Insurance & The Friesland Insurance (DFZ). We followed the Dutch government statement that requires that quality indicators need to be accepted tripartite by patient representatives, healthcare professionals and health insurers.²³ The board evaluated the process and was asked for their approval of the final standard set. The study protocol was approved by the Radboudumc Medical Ethical Committee (Registration # 2017–3154).

Table 1 Steps During the Consensus-Driven RAND/UCLA Modified Delphi Technique

Development Process

The development process included an iterative process in eight consecutive steps: each step provided input for the following step.

Step 1 Literature Search

Potential measures for the standard set were searched in existing guidelines based on the outcome domains. We used a pragmatic explorative approach and did not aim at conducting a full systematic review of the literature. The databases of the Guidelines International Network (G-I-N) and Physical therapy Evidence Database (PEDro) were searched for guideline-based measures for patients with COPD.^24,25 We included all multi- and monodisciplinary Dutch and international clinical practice COPD guidelines for physical therapists, general practitioners and medical specialists. Measures were also searched through websites of organizations developing clinical practice guidelines (see Appendix A). In the next phase, all eligible measures for outcome domains, process and structure measures relevant for patients with COPD were selected. Reasons for exclusion were: not familiar in the Netherlands, not developed and/or not useful for patients with COPD, not primarily recommended in guidelines or not recommended for physical therapy primary care purposes. For each selected measure, the following information was summarized: type of measure (process, structure, outcome); outcome domain (e.g. physical capacity or quality of life); whether it was a PROM, PREM or physical performance measure; content of the measure; time to complete the measure; the minimal clinically important difference (MCID); related measures; whether the measure was already translated in Dutch. We prioritized measures that were already used in daily practice in the Netherlands. In addition, we collected supporting literature about reliability, validity and responsiveness of the measure. To search for methods to categorize patients with COPD in subgroups, the clinical guidelines identified in the literature search were used. Reference checking was performed to search for additional publications.

Step 2 First Online Survey

In total, 37 individuals were invited to participate in an online survey; including physical therapists specialized in COPD (n=25), senior researchers (n=3), policy makers (n=3), representative of a patient association (n=1) and representatives of regional networks of physical therapists specialized in COPD (n=5). Participants were recruited via the contact networks of the advisory board. Each participant needed to have at least 5 years experience in treatment, research or representing patients with COPD. The following two questions were scored for each measure selected from the first step: 1) Is this measure relevant to evaluate the quality of the physical therapy treatment for patients with COPD?; and 2) Is this measure feasible to score at the beginning and the end of the treatment episode for patients with COPD? The selected measures and methods for classifying subgroups were scored using a 9-point Likert scale ranging from totally not relevant/feasible (0) to highly relevant/feasible (9).²⁰ As advised in the RAND/UCLA appropriateness method, the measures that scored a median of ≥7 on relevance and feasibility were initially accepted as measures having added value in daily practice.²⁰ The online survey was performed in LimeSurvey version 2.06.

Step 3 Expert Meeting

To interpret the data of the second step, we formed an expert group consisting of five expert physical therapists (EB, ET, CZ, ML, NP and AH) who also participated in the online survey and had complementary expertise in treatment of patients with COPD, with a mean age of 50 years, and 50% was female. During a face-to-face meeting, the experts reflected on the measures based on supporting literature regarding the validity, reliability and responsiveness of the proposed measures and their own experience. In addition, the experts were asked to interpret, discuss and if needed modify the methods to classify patients in subgroups that were found in the literature search.

Step 4 Patient Interviews

In each of nine purposefully selected physical therapy practices, a patient with COPD was recruited. Purposeful sampling is a strategy in qualitative research to identify and select cases with rich information regarding the subject of interest and is highly appropriate for mixed methods studies.²⁶ Potential patients needed to be 18 years or older, diagnosed with COPD and treated by a physical therapist in the last year. The physical therapist gave the contact details of the patient to the researchers after the patient signed an informed consent form. Two researchers conducted semi-structured interviews (KV and JL) of 30 mins by telephone using an interview guide (see Appendix B). The patients were questioned about their perspectives on relevance and feasibility of outcome domains and proposed measures (PROMs, performance measures and PREMs), and if the measures are useful for the evaluation of physical therapy treatment. Furthermore, we asked patients about their views on relevance and feasibility of the structure, process and outcome measures. All interviews were audio-recorded, transcribed and analyzed using thematic analysis.²⁷ KV and JL independently analyzed the interviews and assigned codes within and between the interviews in quality data software ATLAS.ti 7.0. The assigned codes were compared and sort together in categories and themes.²⁷

Step 5 Consensus Meeting

All 37 individuals of Step 2 were invited for a three-hour face-to-face consensus meeting to select measures for the final standard set. The advisory board was invited to monitor the process. We used the Nominal Group Technique (NGT) to structure the meeting.^28,29 The NGT is a structured brainstorming process and opens the opportunity for all participants to contribute to the discussion.³⁰ First, the results of Step 1 to Step 4 were presented and discussed. Then, the participants voted for inclusion of each initially selected measure and the classification in subgroups by raising their hand through a yes/no vote. The measure was included if ≥80% voted yes for inclusion.³¹ For the measures with a vote of yes between 80% and 60%, alternatives for the proposed measure were discussed with the expert group (Step 6) and re-rated in a second online survey (Step 7). Measures that scored ≤60% were excluded and not considered in the following steps. As part of the NGT, the group had the opportunity to suggest measures that were not initially selected after the literature search.²⁹ These newly suggested measures were rated and those that scored higher than 60% yes of the votes were discussed in the next step with the expert group.

Step 6 Expert and Patient Representative Meeting

The measures for which no consensus (yes votes between 60–80%) were reached and the newly suggested measures (yes votes higher than 60%) in Step 5 were discussed with the expert group, and separately with the Dutch Lung Foundation to obtain additional input from the patient’s perspective. This was an iterative process through a face-to-face expert meeting, followed by consultation by telephone and group discussion via email.

The expert group was asked to suggest eligible alternatives that might be relevant and feasible in daily practice. Furthermore, the expert group developed a guided measurement protocol that provides an overview for physical therapists at what time points the measures need to be completed during the diagnosis and treatment process. Lastly, the expert group was asked to select case-mix variables for the standard set to identify patient characteristics and disease-specific characteristics. When adjusting for these case-mix variables during analysis of treatment outcomes of patients with COPD, interpretation of the standard set is expected to be more accurate.

Step 7 Second Online Survey

The 37 participants of Step 2 were invited for the second online survey. Alternative measures that were suggested by the expert group and the Dutch Lung Foundation were scored on a 9-point Likert scale. The measures were included if they scored a median of ≥7.²⁰

Step 8 Final Approval of the Advisory Board

In the last step, the standard set was presented to the advisory board and if they accepted the standard set as having added value, they were asked to sign an official approval document.

Results

Step 1 Literature Search

After screening nine clinical guidelines^5,15,32–38 and additional literature on reliability and validity,^39–45 45 measures were found and 21 measures were included. The reasons for exclusion of the 24 measures were: not familiar in the Netherlands, not developed and useful for patients with COPD, not primarily advised by guidelines and not for physical therapy primary care purposes. Appendix C shows an overview of all measures. The 21 included measures consisted of one structure measure, two process measures and 18 outcome measures, including 11 PROMs, two PREMs, three physical performance measures and two other described outcome measures derived from a quality indicator. The following ICF outcome domains were selected: physical capacity, muscle function, dyspnea, physical activity and quality of life. Furthermore, we used elements of different clinical guidelines to propose a combined classification method for stratification of patients in subgroups based on the burden of disease; see Appendix D.^5,32–35,37

Step 2 First Online Survey

A total number of 22 out of 37 individuals accepted the invitation (response rate: 60%, mean age 46 years, 47% female), including physical therapists specialized in COPD (n=14), senior researchers (n=3) and representatives of regional networks of physical therapists specialized in COPD (n=5). After analyzing the results of the survey, 7 measures scored a median of ≥7 on relevance and feasibility, and also the proposal to stratify patients in subgroups based on the burden of disease (see Appendix D) scored a median of ≥7. The other measures scored a median ≤6 for relevance and/or feasibility. Table 2 presents all rated measures.

Table 2 Results of Step 2: First Online Survey and Step 5: Consensus Meeting

Step 3 Expert Meeting

The most important statement that the expert group made was that treatment goals of patients with COPD are not only based on improving health condition but also on maintaining health status. They expressed that COPD is a progressive disease with fluctuations in health condition, which influences the expected outcomes of physical therapy treatment. When in further research the standard set is used for the development of quality indicators, caution is required on determining norm values on outcomes of the measures.

In addition, the expert groups suggested further details for the combined classification method in subgroups derived from the literature search. The details were presented at the consensus meeting in Step 5.

Step 4 Patient Interviews

The average age of the nine patients (66% female) who were interviewed was 64.0 years (Standard Deviation [SD] 4.0) and GOLD stage ranged between II (n=1), III (n=3) and IV (n=5). The GOLD stage is a classification of airflow limitation in COPD ranges between I (mild) and IV (very severe).¹⁵ After the data analysis, five main themes emerged: 1) questionnaires, 2) patient satisfaction, 3) number of treatment sessions, 4) clinical practice and 5) quality of care. See Appendix E for all themes, categories and codes. Patients stated that the readability of questionnaires was good. They indicated that they would spend a maximum of 10–15 mins completing questionnaires. Some patients mentioned that patient satisfaction is an element of quality of care. According to the patients, the number of treatment sessions provided by the physical therapist is not a proxy for quality. Some patients stated that an important requirement for quality of treatments is that the practice facility needs to be adequate for doing exercises and should contain helpful equipment like a treadmill, home trainer or leg press. There were different opinions on whether PROMs are relevant for measuring quality. The patients differed in their definition of high quality of care, for example, defined by the treatment effect (maintaining health status), patient-centeredness (communication, the physical therapist listens to me) or being coached by well-educated and specialized physical therapists.

Step 5 Consensus Meeting

In the consensus meeting, 19 individuals were present (response rate: 51%, mean age 43 years and 63% female), including physical therapists specialized in COPD (n=10), senior researchers (n=3), policy makers (n=3), representatives of regional networks of physical therapists specialized in COPD (n=3) and the advisory board (n=5) to monitor the process. Their final votes for inclusion were for five initially selected measures, including: a structure measure with characteristics of practices and physical therapists, and the outcome measures Clinical COPD Questionnaire (CCQ) for quality of life, Medical Research Council Dyspnea (MRC) for dyspnea, Global Perceived Effect (GPE) for experience and the 6-Minute  Walk Test (6-MWT) for physical capacity. All other measures were excluded by the participants.

Some individuals in the group suggested five alternatives for measures that were already derived from the literature search but not selected for the first survey: 1) Adding an additional quality of life questionnaire for patients with a high burden of disease (votes: 92% yes and 8% no); 2) Include treatment costs for value-based healthcare purposes (votes: 57% yes and 43% no); 3) Monitoring physical activity in daily life, as an additional measure that requires further testing in the standard set (votes: 100% yes and 0% no); 4) The Assessment of Burden of COPD tool, also as an additional measure to evaluate whether the questionnaire is useful for evaluating physical therapy interventions (votes: 100% yes and 0% no); 5) The Hand-Held Dynamometer (HHD) (with a Microfet™) for monitoring quadriceps muscle strength (votes: 100% yes and 0% no). All suggested alternative measures were discussed in the next round, except Suggestion 2.

During the meeting, no consensus was reached about the inclusion of the suggested classification in subgroups from the literature. According to the participants, the presented classification method in subgroups was insufficient to determine the prognostic course of the patient group. The group agreed to discuss eligible alternatives with the expert group and patient representatives in the next step. See Table 2 for all votes.

Step 6 Expert and Patient Representative Meeting

In this round, the expert group discussed a) alternatives to classify patients in subgroups, b) how to monitor physical activity in daily life, c) adding a general quality of life questionnaire, d) using HDD (with a Microfet™) for monitoring quadriceps muscle strength, e) development of a guided treatment protocol, and finally, f) selecting case-mix variables:

1. The expert group concluded after an iterative discussion that the described method for classifying subgroups can be replaced by a method that has already been used in the Netherlands, described in the multidisciplinary Dutch Care Standard of the Lung Alliance.³⁴ This guideline was also part of the combined classification method as described in Step 1. The method describes three subgroups classified as light, moderate and high, based on the burden of disease.³⁴ The method is based on cut-off points of the MRC, CCQ and number of exacerbations, lung function and the body mass index (BMI) of the patient.
2. The expert group agreed on monitoring physical activity as an additional measure that requires further testing in the standard set. The physical therapists could choose whether they use a questionnaire, activity diary, accelerometer or other activity trackers for measuring physical activity.
3. The expert group advised that physical therapists could choose one of the three eligible quality of life questionnaires as additional measures in the final standard set. The following questionnaires were suggested: Nijmegen Clinical Screening Instrument (NCSI), St George’s Respiratory Questionnaire (SGRQ) and the COPD Assessment Test (CAT).
4. The expert group agreed on the suggestion of the participants in Step 5 to include the HDD (with a Microfet™) for monitoring quadriceps strength.
5. The expert group was asked to establish a guided measurement protocol that provides an overview for physical therapists at which time points the measures need to be scored during the treatment. They advised to differentiate between mandatory measures for the total patient group, conditional measures depending on the treatment goals and exploratory measures that require further testing. Mandatory measures included the selected structure measure with required characteristics of the practices and the physical therapists, and the outcome measures CCQ, GPE and 6-MWT. The Dutch Lung Foundation preferred to measure the GPE every three months. Conditional measures, only relevant for specific treatment goals, were using the HDD (with a Microfet™) for quadriceps muscle strength, and the MRC for dyspnea. Lastly, exploratory measures were included for monitoring physical activity and the Assessment of Burden of COPD tool.
6. Finally, the expert group chose case-mix variables for a more accurate interpretation of the outcomes of the standard set. The case-mix variables included patient characteristics (age, gender, weight and length) and disease-specific variables (lung values, smoke history, comorbidities, treatment goals and exacerbations). See Table 4 for an explanation of the case-mix variables.

Step 7 Second Online Survey

In this step, 23 individuals (response rate: 64%, mean age 46, 44% female) completed the second online survey, including physical therapists specialized in COPD (n=14), senior researchers (n=3), a policy maker (n=1) and regional networks of physical therapists specialized in COPD (n=5). The alternative method to classify patients in subgroups and measurement protocol scored a median of ≥7 and was therefore included in the final standard set. The alternative suggestions for measuring physical activity and using quality of life questionnaires were scored with a median of 6. Nonetheless, based on narrative suggestions and discussion by the expert group, the accelerometer – when relevant for the treatment goal – can be useful for monitoring physical activity. Therefore, we included the accelerometer as an exploratory measure to monitor the level of physical activity of patients. The additional quality of life questionnaires were excluded. See Table 3 for the final standard set.

Table 3 Final Standard Set

Table 4 Case Mix Variables

Step 8 Final Approval of the Advisory Board

The advisory board accepted the final outcome set as shown in Table 3. All stakeholders signed an official approval document.

Discussion

In this consensus study, a standard set was developed with five outcome domains: physical capacity, muscle strength, physical activity, dyspnea, quality of life, and eight proposed measures were selected for patients with COPD treated in primary care physical therapy practice. The standard set consists of four mandatory measures for all patients with COPD, including one structure measure; characteristics of practices and physical therapists, and three outcome measures; the 6-Minute Walk Test (6-MWT) for physical capacity, the Clinical COPD Questionnaire (CCQ) for quality of life and the Global Perceived Effect (GPE) for experience. Two conditional measures are included depending on the treatment goal: using the HDD (with a Microfet™) for quadriceps muscle strength and Medical Research Council Dyspnea (MRC) for monitoring dyspnea. Two exploratory measures are included: use of an accelerometer for monitoring physical activity in daily life and The Assessment of Burden of COPD tool (ABC). For identifying subgroups based on the burden of disease, the method of the Dutch care standard of the Lung Alliance is included. Finally, case mix variables were selected for a more accurate interpretation of the outcomes in the standard set.

The standard set was accepted as having expected added value in clinical practice and is therefore deemed useful for the interaction between a patient and a physical therapist. Furthermore, comparison of outcomes of the standard set between physical therapists on individual and group level opens the opportunity to learn from routine data collection, and finally the standard set provides a basis for the development of quality indicators. To our knowledge, this is the first study that describes a standard set for patients with COPD for these specific goals including development of quality indicators in primary care physical therapy.

The recent study of Souto-Miranda et al described domains for measures in a core outcome set based on stakeholders perspectives (patients, informal care providers and health professionals).⁴⁶ Important described needs (e.g. improving exercise tolerance and reducing dyspnea) by the stakeholders are in line with the measures in the standard set (e.g. 6-MWT and MRC).⁴⁶ However, there are also differences between the study of Souto-Miranda et al and this study. The goal of our standard set is developing quality indicators and enhance quality improvement initiatives in clinical practice. The described goals of the study of Souto-Miranda et al are to inform on a core set that generates consistency among clinical trials and decrease risk of bias in research studies by standardizing outcomes.

The expert group stated in Step 3 that patients with COPD are a heterogeneous patient group with more or less comorbidities and exacerbations that cannot always be influenced by physical therapists. They expressed that for that reason caution is required with interpreting outcomes between physical therapists and practices. We therefore included case-mix variables and a proposal to stratify patients in subgroups to allocate patients in more homogeneous subgroups. Identifying these subgroups opens the opportunity to compare and predict outcomes more accurate for the same patient population. Using the standard set combined with stratification in subgroups and case-mix variables opens the opportunity for physical therapists and practices to use quality indicators as a learning tool for quality improvement initiatives by comparing outcomes between their peers.^19,47 Nonetheless, when comparing and interpreting outcomes of physiotherapists and practices, the fluctuating health condition of patients with COPD needs to be included.⁴⁸

As described in the introduction, stakeholder engagement is highly important in the development of quality indicators. Therefore, we were pleased that all stakeholders were included in the consensus rounds. Zorgverzekeraars Nederland (ZN),⁴⁹ which is the umbrella organization of the ten health insurance companies in the Netherlands, accepted the outcomes of the final standard set and agreed to use the standard set in the development of quality indicators.

Our study has several limitations. One limitation of our study is that the consensus rounds were conducted in the Netherlands and focused on the Dutch healthcare system. In addition, we selected outcome domains based on a Dutch guideline for physical therapists, and preferred proposed measures that were already used in the Netherlands. The generalizability for international use of the standard set of outcome domains and proposed measures may therefore be limited. However, the selection of measures was based on a literature review of international clinical practice guidelines. This is in line with other studies developing quality indicators, e.g. the study of Westby 2018 et al.⁵⁰ Still, the context of each country needs to be taken into account, and we think the focused method of our study was helpful to encourage successful implementation in Dutch physical therapy practice.⁵¹

The COMET initiative provides guidance for the selection of outcome domains and outcome measures in developing core outcome sets.⁵² The COMET initiative is a valuable and important initiative to develop and inform on core outcome sets (COS) for clinical trial purposes and clinical auditing. We chose to use the RAND/UCLA appropriateness method, which is widely used and provides a manual for synthesizing expert opinion and evidence for the development of quality indicators.^11,47,53 The steps described by COMET are to a large extent similar with the RAND/UCLA appropriateness method, including literature review, Delphi procedure and face-to-face consensus meetings; the purpose though is different. In addition, the COMET handbook version 1.0 was published after conduction of our study.⁵²

After this consensus study, the standard set was implemented for a pilot test in the Netherlands for the development of quality indicators. In January 2018, a large pilot was launched where over 250 physical therapists started using the standard set for collecting data of approximately 4000 patients with COPD treated in primary care. In the Netherlands, many software systems for electronic health records (EHR) are used in primary care. The EHRs must be connected to a secure database. Also, the data collection of all EHRs needs to be standardized with the standard set; otherwise, comparison of the outcomes can be invalid. An important requirement for successful data collection of the standard set is that the infrastructure is adequate.

This study presents a standard set of outcome domains and proposed measures for patients with COPD in primary care physical therapy; each measure is accepted for relevance and feasibility by stakeholders. The standard set is a promising basis for the development of quality indicators in primary care physical therapy practice.