Measuring Adherence to Pulmonary Rehabilitation: A Prospective Validation Study of the Dutch Version of the Rehabilitation Adherence Measure for Athletic Training (RAdMAT-NL)

Plain Language Summary

Chronic obstructive pulmonary disease, COPD, is a common chronic disease and is treated with medication and exercises. To be beneficial, it is important to adhere to the exercise program prescribed by the physiotherapist or rehabilitation doctor. Research in the field has introduced various tools to measure this adherence. One such tool is the Rehabilitation Adherence Measure for Athletic Training, or RAdMAT for short. The results of these studies demonstrated that the RAdMAT has good psychometric properties in physiotherapy in general. In the Netherlands, a Dutch version of this questionnaire, known as the RAdMAT-NL, is available to rate adherence to prescribed exercises. The quality of these ratings – that is, the reliability and validity of the RAdMAT-NL - is unknown. Our study aimed to explore the structure and accuracy of the RAdMAT-NL in patients with COPD.

Almost 200 patients with COPD who received an exercise program from their physiotherapist in the Netherlands and Belgium participated in our study. For three months, these patients and their physiotherapist provided information about the exercises, including the scores given to patients on the RAdMAT-NL. By analyzing these scores, we found that the RAdMAT-NL consists of two subscales, “Participation” with 13 items and “Communication” with 3 items. The 13 items of Participation may be summed to a single score of adherence to the exercises that may help the doctor or physiotherapist in improving adherence to the exercises if needed.

Introduction

Physical rehabilitation involves self-management, and requires effort from patients themselves in following prescribed exercises at home. Adherence to exercise is therefore essential to enhance rehabilitation outcomes and to improve patients’ health and quality of life.¹ Adherence is important in many fields of healthcare and patients who fail to adhere, may experience prolonged treatment and less favorable outcomes.² To keep healthcare affordable and for improving patient outcomes, attention must be paid to adherence.³

Adherence is of particular importance in chronic obstructive pulmonary disease (COPD). Fewer than half of therapies for COPD are taken as prescribed.⁴ Medication adherence can be monitored by eg, confirming drug use and dose or pill counts.⁵ In rehabilitation, the assessment of exercise adherence is less straightforward because prescribed exercises largely take place at home, with patients and their caregivers making decisions as to whether exercises should be started or continued, often without consulting their healthcare provider.⁶ As a result, it is not always clear to professionals whether patients are adherent or not. Professionals tend to make their own judgements on patients’ adherence by simply asking or by observing treatment progress.⁷ The accuracy of their judgements however is uncertain. Therefore, standardized instruments are available to rate the extent and reasons for non-adherence.⁷ On the basis of early signals, the treatment might be adapted to the specific needs of a patient or measures can be taken for improvement.⁸

Literature shows that instruments are available to measure adherence to exercise in a rehabilitation setting.⁹ Two appear to be valid and reliable: the Sport Injury Rehabilitation Adherence Scale (SIRAS)¹⁰ and the Rehabilitation Adherence Measure for Athletic Training (RAdMAT).¹¹ The SIRAS, with only three items, has been criticized for its limited ability to adequately capture the various behavioral aspects important for adherence.¹¹ The RAdMAT allows for a more detailed and comprehensive assessment of adherence and includes 16 items on patients’ attitudes, communication and clinic behaviors. The RAdMAT has been examined within a broader physiotherapy setting in patients with shoulder problems and showed good psychometric properties.¹² Previous validation (dimensionality and construct validity) of the RAdMAT recommended the use of three subscales, but suggested that a single score for overall adherence also might be calculated.^10,11 Based on the acceptability of the RAdMAT, a Dutch version (RAdMAT-NL) was designed to be used for the Dutch-speaking population. This version was prepared by a native speaker based on the guidelines of translating questionnaires.¹³ The RAdMAT-NL showed a high inter-rater reliability in patients from a primary physiotherapy setting.¹⁴ Therefore, the RAdMAT-NL seems to be appropriate for quantifying adherence in patients with COPD. However, the validity of the RAdMAT-NL, especially its dimensionality and construct validity, is unknown.

The aim of this study was to evaluate the dimensionality and construct validity of the RAdMAT-NL in patients with chronic obstructive pulmonary disease undertaking pulmonary rehabilitation in a primary physiotherapy practice. Secondly, we examined if the items of the RAdMAT-NL could be summed to a single score of adherence.

Methods

The reporting of this study complies with the Strengthening The Reporting of Observational Studies in Epidemiology (STROBE) checklist for cohort studies.¹⁵

Participants

Participants of this prospective study were Dutch/Flemish-speaking patients aged ≥18 years who were recruited between January 2021 and December 2021 from 53 primary physiotherapy practices and from the COPD patient organizations from the Netherlands and Belgium. Included were patients with COPD, with airflow limitation stage Global Initiative for Chronic Obstructive Lung Disease (GOLD) II–IV¹⁶ and having rehabilitation sessions for at least once a month, for at least one month. The exclusion criteria were home-based rehabilitation and insufficient mastery of the Dutch/Flemish language to complete the questionnaires.

Physiotherapy practices were approached by email and social media for participation and patients were recruited by their attending physiotherapist. Patients who were willing to participate were contacted by the researcher per email for further information on their participation, and to obtain informed consent. For patients without an email address, this was done by their physiotherapist at their visit to the physiotherapy practice. Patients were also recruited via the COPD patient organization in the Netherlands and Belgium who, in turn, invited their physiotherapist for participation.

The study complies with the Declaration of Helsinki and was approved by the Ethical Committee Psychology of the University of Groningen (PSY-1920-S-0504).

Procedures

Before the start of the study, all participating physiotherapists received an email about the implementation of the study, including a manual on how to use and administer the RAdMAT-NL, the SIRAS and the Five Times Sit to Stand Test (FTSST).

After signing informed consent and inclusion, patients completed an online (Qualtrics; online survey software)¹⁷ or paper form on their demographic characteristics; age (years), gender (male/female), level of education (lower/middle/high), smoking status (never/stopped/still smoking). If applicable, a paper form was sent by the researcher to the participating physiotherapy practice for completion.

At the same time, the physiotherapist provided information on the characteristics of the disease; GOLD classification as determined by the pulmonologist (II/III/IV), Medical Research Council (MRC) dyspnea score¹⁸ as determined by the patient (0/1/2/3/4/5), time since onset of COPD (years), duration of physiotherapeutic rehabilitation (0–3 months/3-6 months/6-12 months/1-5 years/>5 years), and administered the FTSST to the patient. The patients provided information about their homework exercises and their health at one, two and three months after inclusion. After three months, the physiotherapist provided information and measurements on the patient’s rehabilitation.

Assessment Instruments

RAdMAT-NL

The RAdMAT-NL evaluates clinic-based adherence with 16-items using a four-point rating scale (never = 1, always = 4).¹⁴ The original RAdMAT consists of three subscales: Attendance/participation “follows the prescribed rehabilitation plan” (items 1–5, range 5–20 points), Communication “asks questions about his/her rehabilitation” (items 6–8, range 3–12 points), and Attitude/effort “gives 100% effort in rehabilitation sessions” (items 9–16, range 8–32 points). The total scale range is 16–64 points (maximal adherence). The RAdMAT-NL was completed by the physiotherapist after three months of rehabilitation, independent of the patient and not in their presence.

Sport Injury Rehabilitation Adherence Scale (SIRAS)

The SIRAS rates, on a five-point scale, adherence to clinic-based rehabilitation, using three items on amount of effort taken, on extent of following instructions and advice, and on receptiveness to change in the rehabilitation program.¹² Scores may range between 3 and 15 points (maximal adherence).¹⁰ Like the RAdMAT-NL, the SIRAS was completed by the physiotherapist after three months of rehabilitation independent of the patient, and not in their presence.

Five Times Sit to Stand Test (FTSST)

Measuring exercise skills, eg, lower extremity strength, may be a way to assess adherence.¹⁹ If patients adhere to their rehabilitation program, their muscle strength will increase (or is at least stable) over time.¹⁹ The FTSST is a test of lower limb function to assess the fastest time taken to stand five times from a chair with arms folded.²⁰ The FTSST was assessed by the physiotherapist during consultation two times, first after signing informed consent, and second after three months of rehabilitation.

Home-Based Adherence Form

Patients completed a form with four items, to rate the extent they had performed their prescribed exercises the past month²¹ using a five-point rating scale (as often as I had to = 1, not performed = 5).

The home-based adherence form was filled out by the patient after one, two and three months of rehabilitation. Patients could also indicate whether or not they had experienced a COVID-19 infection or a lung attack²² in that month. If so, those data were excluded from the psychometric analysis of the RAdMAT-NL.

Perceived Adherence and Percentage of Attendance

Perceived adherence and percentage of attendance were rated by patients after three months of rehabilitation. Perceived adherence (home-based and clinic-based) included the extent to which they succeeded to attend appointments, performed prescribed exercises, and the extent to which they asked questions and gave feedback about their rehabilitation. Adherence was rated on a five-point scale (did not succeed at all = 1, totally succeeded = 5).¹⁹

The percentage of attendance was calculated for each patient by the number of appointments attended divided by the number of appointments scheduled, multiplied by 100.¹²

Statistical Analysis

Data were analyzed using R version 4.0.3.²³ using the {psych} and {eRm} package.

Only complete datasets were included in the analysis. Descriptive statistics were used to summarize the baseline demographic and clinical characteristics of the patients. Variables were expressed in percentages or as the median with interquartile range (IQR).

The dimensionality of the RAdMAT-NL ie, the number of dimensions present in the RAdMAT-NL itemset, was evaluated with Horn’s parallel analysis.²⁴ Parallel analysis is a technique that works by creating a random dataset with the same numbers of observations and variables as the original data. The associated scree plots based on the actual and random data allow to distinguish true factors from factors that are obtained by chance.²⁴ Based on the results of the parallel analysis, a forced exploratory factor analysis (EFA, principal axis factoring) was performed. Specificity of the extracted factors was evaluated by the absence of cross-loading items (>0.45) using the cut-offs according to sample size outlined by Hair.²⁵ Fit of the factor model was assessed with the root mean square residuals (RMSR, ≤0.05), the Fit based upon off diagonal values (>0.95), and the Tucker Lewis Index (TLI, >0.90).²⁶ Prior to performing the EFA, the appropriateness of the data for factor analysis was assessed with the Kaiser–Meyer–Olkin measure of sampling adequacy (KMO, >0.70) and Bartlett’s test of sphericity (p < 0.05).²⁴

Subsequently, essential unidimensionality of the extracted RAdMAT-NL subscales was tested with the one parameter logistic Rasch model for binary item responses. The Rasch model is a formula describing the probability of an item response (0 or 1) as a function of trait level.²⁷ The basic assumption is that with increasing trait levels an increasingly higher probability of a positive score is expected; 1 instead of 0 for a binary scored item.²⁷ Rasch analysis estimates item measures (difficulty to endorse) and person trait measures which are both expressed on a common logit unit scale. To facilitate the Rasch model analysis, the item scores were dichotomized according to their mean (item score < mean = 0; item score > mean = 1). Item fit to the Rasch model was examined by mean square infit and outfit effect sizes (ranges 0.5 to 1.5 were considered acceptable).²⁸

Construct validity of the RAdMAT-NL subscales was examined by calculating Spearman correlations with; SIRAS, percentage change on FTSST, self-reported home-based adherence, overall adherence and percentage attendance at scheduled rehabilitation appointments. The strength of correlations was interpreted based on the guidelines described by Cohen: small = 0.10–0.29, medium = 0.30–0.49 and large is 0.50–1.0.²⁹

Results

Patients from 53 different physiotherapy practices participated in the study. Out of 199 patients who gave informed consent, data from 193 patients were analyzed (Additional File 1). The most common reasons for exclusion were; the physiotherapists did not provide any information, or the patient left the study (died or diagnosed with cancer).

Table 1 summarizes the demographic and disease characteristics of the patients. Characteristics of the participating physiotherapists are summarized in Additional File 2.

Table 1 Demographic and Disease Characteristics Patients (n = 193)

Dimensionality of the RAdMAT-NL

The KMO was 0.90, and all values for individual items were >0.76, exceeding the recommended minimum value of 0.7. Bartlett’s test of sphericity was, χ² (120) = 1759.6, p < 0.001, below the needed p < 0.0523, indicating the appropriateness of conducting a factor analysis.

The parallel analysis indicated the presence of two dimensions emerging over chance (Figure 1). The blue line with the crosses (the factor eigenvalues) represents the original data scree plot. The red dashed line represents a scree plot based on the random data. The figure shows that eigen value of the second large factor is slightly above that obtained by chance. Table 2 shows the factor loadings of the subsequent forced two-factor model. The factors were labelled as Participation (13 items) and Communication (3 items) and explained 44% and 7% of the variance respectively. There were no cross-loadings, items loading >0.45 on both factors. Model fit statistics were within acceptable ranges (RMSR of 0.06, a Fit based upon off diagonal of 0.98, a TLI of 0.803).

Table 2 Item Frequency Distribution, Factor Loadings, Eigenvalues and Percentage of Explained Variance for the Two Factors of the RAdMAT-NL (n = 193)

Figure 1 Parallel analysis.

The RAdMAT-NL total scale and subscales (“Participation” and “Communication”) were subjected to the unidimensional Rasch model (Table 3). Both subscales showed adequate fit to the unidimensional model, except for item 14 from the Participation subscale where infit and outfit statistics fell slightly above the 1.5 cutoff. For the RAdMAT-NL total scale, the Rasch model was rejected. In line with the extracted subscales, Communication items 6 to 8 interfered with unidimensionality.

Table 3 Item Fit Statistics for Participation and Communication Scales and Total RAdMAT-NL

Construct Validity

The Spearman correlation coefficients between the subscales of the RAdMAT-NL and the other indicators or measures of adherence, SIRAS, percentage change on FTSST, self-reported home-based adherence, overall adherence and percentage attendance at scheduled rehabilitation appointments are presented in Table 4. As expected, the RAdMAT-NL subscale participation was positively associated with all other measures indicative of adherence. Self-reported home-based adherence had a medium positive correlation, all other measures had a large positive correlation with the RAdMAT-NL subscale Participation, demonstrating evidence for construct (convergent) validity.

Table 4 Construct Validity; Correlations with FTSST Change and Adherence Measures

Discussion

This study assessed the dimensionality and the construct validity of the Dutch version of the Rehabilitation Adherence Measure for Athletic Training (RAdMAT-NL) in patients with COPD undertaking pulmonary rehabilitation in a primary physiotherapy practice in the Netherlands and Belgium. The study also examined whether the 16 items of the RAdMAT-NL could be summed to a single score of adherence. The RAdMAT-NL exhibited two subscales (Participation and Communication) that fitted the unidimensional Rasch model for objective measurement. Construct validity was supported by convergence with other established measures of adherence.

Dimensionality of the RAdMAT-NL

The parallel analysis results of this study show a different number of dimensions present in the RAdMAT-NL compared to the original research;¹¹ two factors instead of three factors. Factor 2, Communication, is the same factor as in the original research, however Factor 1 and 3 of the study of Granquist et al¹¹ are reflected in our study as one factor, Participation. Fit statistics indicated a good fit for the two-factor solution. Rasch model fit statistics indicated unidimensionality for both dimensions. Only Participation item 14 had infit and outfit statistics slightly above the 1.5 cut-off, interfering with unidimensionality but not degrading for measurement.²⁸

Granquist et al suggested the RAdMAT could be used as a single score but did not statistically show this.¹⁰ Since it may be convenient for research, in practice and in evaluation research to use a single sum score for adherence, it was tested whether this is allowed. The unidimensional Rasch model was rejected for a summed score of all 16 RAdMAT-NL items because of the Communication items. Granquist et al¹¹ also made their own observation regarding the Communication items. They described that two previous studies^30,31 addressed communication as an important factor for rehabilitation adherence, but that literature does not identify communication as a rehabilitation-adherence behavior.³² Granquist et al¹¹ concluded that athletic trainers likely use other cues/communication and that this should be kept in mind when interpreting the results of adherence. Generally, we observed more score variation on the Communication items than on the Participation items (with the exception of item 16). The presence of more variation could indicate a wider range of opinions or perspectives among the physiotherapists. Although communication is an important aspect of rehabilitation, the Communication items involve the patient’s communication only, which is less appropriate for assessment of adherence in a medical rehabilitation setting. Items evaluating the quality of patient-therapist communication are probably more appropriate in a medical rehabilitation setting, where shared decision-making is central.

Construct Validity

In line with the results of Granquist et al¹¹ and Clark et al,¹² our results demonstrated large correlations between the RAdMAT-NL subscale Participation and SIRAS and between the RAdMAT-NL subscale Participation and percentage attendance. Furthermore, this study supported convergent validity by a medium correlation between the subscale Participation and self-reported home-based adherence, and large correlations with perceived overall adherence and the change on the FTSST. The overall observed positive correlations confirm that the RAdMAT-NL subscale Participation is associated with attendance to appointments, exercise skills and self-rated adherence which suggests the potential clinical relevance of the RAdMAT-NL.

Strengths and Limitations

The strengths of this study are inclusion of patients from 53 physiotherapy practices and from patient-organizations in The Netherlands and Belgium, which enhances the generalizability of the study results. Another strength is the use of parallel analysis; parallel analysis is considered one of the best options for determining the factors to retain in factor analysis because it provides a statistically rigorous approach. It addresses the issue of overestimating the number of factors to retain, which is a common challenge in exploratory factor analysis.²⁴ Finally, a strength of this study is that prospective data were collected in a real-life setting of primary physiotherapy practices, increasing ecological validity.³³

This study also has some limitations. This study was conducted during the COVID-19 pandemic and may have affected motivation of patients for physiotherapy and the way physiotherapists work. Counselling may not always have been able to take place physically, but also online and may have influenced the Communication items.

Follow-up research should further explore the validity of the RAdMAT-NL, preferably with a larger sample size, to assess the robustness of the results and especially to further investigate content validity with regard to the formulation of the Communication items and whether or not to retain Communication items to assess adherence in medical rehabilitation.

Implications

No Dutch language instrument for exercise adherence was available. The RAdMAT-NL in its current form could be used with some caution as an interim assessment instrument of exercise adherence for patients who are not progressing as expected. Low scores on the subscales of the RAdMAT-NL may indicate the need for interventions to enhance patient’s adherence. According to the American College of Sports Medicine (ACSM) guidelines, a score of at least 85% must be achieved to be adherent to the rehabilitation program.³⁴ The RAdMAT-NL may also be used or studied in relation to the patient psychological determinants of adherence, such as motivation and perceived control. That is, while the RAdMAT-NL provides the professional’s perspective on the patient’s adherence, the psychological determinants may provide the patient perspective, and together they may predict adherence even better. To investigate whether this is indeed possible, we used the RAdMAT-NL in combination with the Theory of Planned Behavior, among others, to develop a prediction model for adherence.³⁵

Conclusions

The RAdMAT-NL, with two subscales (Participation and Communication) and good construct validity, can be used to quantify exercise adherence in patients with COPD undertaking pulmonary rehabilitation in a primary physiotherapy practice. With only 16 items in a simple format, the RAdMAT-NL is also relatively easy to use, whether by healthcare providers in clinical practice or for research purposes.