Improving the response rate of patient-reported outcome measures in an Australian tertiary metropolitan hospital

Key points for decision makers

• Routine collection of patient-reported outcome measures (PROMs) is clinically important for the ongoing care of patients.
• The low response rate in the collection of PROMs is one of the major hurdles toachieving this.
• Response rate is highly variable but can be dramatically improved through implement-ing a variety of techniques.

Introduction

PROMs are objective questionnaires used to measure quality of life data.^1–4 Many PROMs are now considered the gold standard in measuring outcomes in orthopedics^5,6 and their use improves both patient satisfaction and clinical practice.^4,7 New Zealand, Sweden, and the UK have been leading the world in the collection of routine PROMs for joint registries, paving the way for quality of life data to be analyzed nationally.^1,8–10 Despite the benefits associated with the collection of PROMs,^11,12,21 its uptake into routine clinical practice around the rest of the world remains slow.¹

There are many challenges to the routine collection of PROMs in the public hospital system^1,2,10 including the time, effort and cost required.¹³ Although the development of a web-based repository for PROMs can help tackle some of these challenges,^14,15 the utilization of this method remains limited within the orthopedic patient population.^1,16,17 Another challenge is the low response rate, which is often fueled by disinterest, lack of time or inability to comprehend the questions.¹⁸ Certain characteristics that hinder the successful completion of PROMs include age greater than 75 years, social economic status, living alone, poorer pre-operative health, Hispanic or Black ethnicity, private health insurance and participants who have had previous surgeries.^11,18

There is currently a lack of literature on overcoming the ongoing challenge of low response rate to patient outcome collection. The purpose of this study is to assess the predictors of completion of PROMs in the orthopedic outpatient clinic at an Australian public hospital and determine what strategies lead to an improvement in PROM response.

Methods

A prospective cohort study was conducted in an orthopedic outpatient setting of a metropolitan tertiary hospital assessing the response rate of PROM collection. Data collection occurred from February to June 2016. All patients attending the outpatient clinic were invited to complete a PROM relevant to their presenting condition via paper or digital format. All patients were asked to complete two generic wellbeing and condition-specific PROMs relevant to their presenting condition and operation (Table 1).

Table 1 Condition-specific PROMs for common operations

The primary outcome was the response rate, which was defined as fully completing and returning the PROMs back to the investigators. Data collected included patient demographics (age, sex, primary language spoken) and visit outcome. All patients were able to opt out of filling in a PROM and had their reason recorded for analysis. During the study period, eight different methods or environmental variations were employed and the response rate recorded (Table 2).

Table 2 The eight different interventions trialed throughout the study period

Data analysis

A bivariate analysis was performed on the demographic data (Table 3). Age and time spent were reviewed as continuous variables; sex (male vs female), language (English vs non-English speaking), PROM pathology (upper vs lower limb) and the PROM type (visual analog scales vs Likert scales) as binary variables; and visit outcome as nominal categories (pre-, post-, non-op). Continuous variables were presented as means/standard deviations, and analyzed with simple logistic regression analysis. Categorical variables were presented as absolute numbers/frequency and analyzed using a chi-squared/Fisher’s exact analysis. Those that had significance of p<0.05 were retained for model building. Collinearity was assessed between variables and one factor (PROM questionnaire) was found to be strongly correlated R ≥0.5 so removed from the multivariate analysis. A manual backward stepwise regression was used to develop a multivariate logistic regression model of predictors of response rate of PROMs. A mixed effect analysis was used to assess for any unmeasured influences by repeat observations of Cohort H over 9 weeks of intervention, and the effect applied to the other cohorts. Only factors significant at the α =0.05 level were retained in the final model, except age and sex which were considered as potential confounders. The results of the multivariate analysis (Table 4) were reported with ORs and 95% CIs. Analysis was completed in Stata v14.1 with collin commands used to assess collinearity.

Table 3 Bivariate analysis

Table 3

Table 4 Multivariate analysis

Results

Bivariate analysis

Over the course of the research period, 2,338 patients were seen; of these, 414 were excluded and 1,924 patients used for analysis. Of the analyzed patients, 47.1% were males (mean age 51.16, s.d. 18.79) and 52.9% females (mean age 58.14, s.d. 17.29) (Figure 2). Gender was not found to have a significant effect on response rates between males and females (χ<0.03, p=0.87). Increasing age was found to adversely affect rate of response with a significance of (p=0.002, OR 0.99, 95% CI: 0.987, 0.997) with a significant drop in response between patients aged 60–69 compared with 50–59 (p=0.011, OR 0.69, 95% CI: 0.514, 0.918). Of the 40 different languages spoken by patients attending clinic (Table 5), English-speaking patients completed a PROM 67.18% of the time, compared with non-English speaking patients who completed it 40.05% of the time (χ²=95.78, p<0.01). New referral patients had a response rate of 68.8%, review patients with non-operative management were least likely at a rate of 44.9% and review patients who were already on a surgical waiting list had a response rate of 64.21%. The differences in response rate were found to be significant between these groups (χ=71.28, p<0.01). There was no significant difference in response rate between patients who had an upper limb pathology (61.08%) and a lower limb pathology (61.29%) (χ=0.008, p=0.93). There was no difference found in response rate between PROMs utilizing Likert scales (60.86%) and those utilizing a visual analog scale (62.73%) (χ=0.44, p=0.506). The response rate based on specific PROMs was found in patients completing the Western Ontario Shoulder Instability Index (75%), International Hip Outcome Tool (67.27%), Oxford Shoulder Score (66.67%), Manchester-Oxford Foot Score (64.39%), Oxford Hip Score (60.57%), Oxford Knee Score (59.47%), Disabilities of the Arm, Shoulder and Hand Score (58.44%) and Western Ontario Rotator Cuff Index (58.13%). However, these differences were not statistically significant (χ=11.4, p=0.122).

Table 5 List of languages spoken by patients

Figure 2 Gender age distribution.

We found that the highest response rate was in Cohort H (81.01%); this was followed by Cohort G (65.45%), Cohort E (62.79%), Cohort D (56.25%), Cohort F (54.89%), Cohort C (36.94%), Cohort B (21.11%) and Cohort A (13.33%) (Figure 1). The difference in response rate between Cohort H and the other variables were significant (χ=389, p<0.01).

Figure 1 Response rates for all eight cohorts.Notes: Cohort A: verbally invited to complete a PROM after seeing their surgeon. Cohort B: verbally invited to complete a PROM prior to seeing their surgeon in the waiting room. Cohort C: approached and verbally invited to complete a PROM prior to seeing their surgeon. Cohort D: approached, given an instruction sheet that explained the research and verbally invited to complete a PROM prior to seeing their surgeon. Cohort E: received an instruction sheet, asking to complete a PROM as part of routine care prior to seeing their surgeon. Cohort F: received an instruction sheet, asking to complete a PROM as part of routine care prior to seeing their surgeon and instructed that the consulting surgeon would be assessing the form. Cohort G: a registration desk was set up where patients were given an instruction sheet, asked to complete the PROM as part of routine care and return afterwards. Cohort H: instructed to line up at the registration desk where they were given an instruction sheet, asked to complete the PROM as part of routine care and return afterwards.

There were multiple reasons why a patient did not complete a PROM. These included patients leaving the waiting room to (30%), requiring an interpreter (27%), recently completing the same PROM (19%), patients who had other physical or medical impairment not related to their orthopedic pathology that would confound the PROM results (8%), refusal (4%), missed patients (2%), vision impairment (1%), poor literacy (0.8%), incomplete PROMs (0.2%), extreme pain (0.1%), and being in handcuffs (0.1%).

Multivariate analysis

There was no difference found between the naive model and the mixed effect model (Table 4). Overall the mixed-effect logistic regression model found that there was a significant difference (p<0.01) between the eight cohorts after including the independent variables into the model. Of these variables, there were multiple predictors that were found to be associated with a better response rate. These predictors were: being younger (OR=0.99, 95% CI: 0.984, 0.996); having a longer wait time (OR=1.001, 95% CI: 1.00, 1.002); being of English-speaking background (OR=3.758, 95% CI: 2.854, 4.949); being a new patient (OR=2.227, 95% CI: 1.662, 2.984) or post-operative patient (OR=1.738, 95% CI: 1.299, 2.324) compared with non-operative patients; and having the intervention performed on Cohort H compared with Cohort A (OR=0.0296, 95% CI: 0.018, 0.05), Cohort B (OR=0.0.058, 95% CI: 0.033, 0.105), Cohort C (OR=0.113, 95% CI: 0.076, 0.168), Cohort D (OR=0.334, 95% CI: 0.240, 0.465), Cohort E (OR=0.392, 95% CI: 0.251, 0.611) and Cohort F (OR=0.235, 95% CI: 0.164, 0.337).

Discussion

The benefits of routine collection of PROMs are well documented.^1,12,19 They improve patient satisfaction, enable early screening of functional decline and are extremely informative for clinicians.^4,7,20,21 PROMs improve patient consultation and can be used for national quality control and population-based research.^8,22,23 However, poor response rate continues to be a challenge that hinders its routine collection.^11,18

The findings of this study show that the response rate can be significantly improved through the use of different methods or environmental variations. The most successful approach was found in Cohort H utilizing strategies including collecting PROMs prior to clinic, approaching patients individually, directing patients through an administrative area, distributing an information sheet and informing patients that the collection of PROMs was part of routine care. This approach had a response rate of 81.35%, which compares favorably with other reported PROM collection registries – UK National Joint Registry (64%), the Swedish Hip Arthroplasty Registry (79%), and the New Zealand Joint Registry (70%).^8,10,24–26 We found that making PROM collection a part of routine care and the development of a specific PROM administration desk yielded the greatest improvement in response rate. We found that the response rate dropped when patients were asked to take their completed PROM into the consulting room to be assessed by the surgeon (p=0.08, OR 0.64, 95% CI 0.389, 1.058). This finding reflects the logistical difficulty of PROM collection when a multitude of users are required to participate in the collection process.

Currently, there are multiple known predictors to poor response rates described in the literature.^1,11,18,27 Our study reinforces some of these predictors, including age and language.¹⁸ Other predictors such as a shorter length of time in the waiting room and review patients for non-operative treatment were also found to reduce the response rate to PROMs. Visual analog scales have been demonstrated as the most accurate measure for disabilities.²⁵ Our study also found no difference in response rates between PROMs which utilized visual analog scales and Likert scales.

Overall the results of our study have identified factors that can be broadly split into modifiable and non-modifiable factors. The modifiable factors, such as the environmental collection modifications demonstrated in our study, were found to sequentially improve the collection response rate. Non-modifiable factors include age, language, literacy and computer skills, current stage within the surgical journey, vision impairment and patient time constraints. Measures to overcome these factors include the availability of paper format, using multi-lingual PROMs, staff availability to assist with vision impairment, literacy and computer skills, and providing information regarding the purpose of PROM collection and utilizing the patients waiting time in outpatients to collect PROMs.

This study aims to describe steps to improve the significant challenge of response rate in PROM collection. Future routine collection of PROMs will enhance the patient experience as well as for population-level quality assurance and research.^2,28

Limitations

There are a number of limitations that can be identified with this study. Although this was a prospective cohort study, there was no randomization. Allocation of patients into cohorts presents potential confounding variables, including temporal bias, which could be overcome by repeating each collection method over multiple clinics or randomization of patients to different methods of collection during a clinic. However, this would be logistically infeasible at our institution. Participants of this study are a small subset of orthopedic patients at a large metropolitan public hospital with particular socioeconomic, education and language variables and as such the results may not be generalized to dissimilar populations.

The future

The future of the collection of PROMs is promising and following completion of this study, our goal is to implement the routine collection of PROMs with the lessons learned from our research. Cost and efficiency can be improved by the use of a web-based repository system to automate much of this process, requiring only the application and maintenance of the program.¹⁴ Such a system has been developed at our institution¹⁴ and successful implementation is the next major challenge that we face.

Conclusion

One of the major barriers to the routine collection of PROMs in the outpatient setting is the low patient response rate, and this study identifies several predictors that affect successful completion. Known predictors such as age and language were reconfirmed and others such as waiting time in clinic, new referral patients, and pre-operative patients were identified. This study also demonstrates that a high response rate can be achieved by alterations in the method of collection.