Improving the response rate of patient-reported outcome measures in an Australian tertiary metropolitan hospital
Received 14 January 2018
Accepted for publication 30 January 2019
Published 10 July 2019 Volume 2019:10 Pages 217—226
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 2
Editor who approved publication: Professor Lynne Nemeth
Andy Ho1,2, Christa Purdie1,2, Oren Tirosh,3 Phong Tran3,4
1Faculty of Medicine, University of Melbourne, Melbourne, VIC, Australia; 2Western Clinical School, Western Health, Footscray, Melbourne, VIC, Australia; 3Department of Orthopaedic Surgery, Western Health, Melbourne, VIC, Australia; 4Australian Institute for Musculoskeletal Science (AIMSS), The University of Melbourne and Western Health, St. Albans, VIC, Australia
Background: The benefits of collecting patient-reported outcome measures (PROMs) for clinical care are widely accepted; however, the collection and response rate remain a significant barrier.
Objective: The objective of this study was to assess predictors of successful PROM response rate in an orthopedic outpatient setting at a public tertiary hospital.
Method: A prospective cohort study was conducted at a metropolitan hospital assessing the response rate after a number of interventions in the collection of PROMs in the orthopedic outpatient setting. All patients were invited to complete a PROM relevant to their presenting condition. Eight cohorts were studied, all different in the process of collection, the timing of collection and the physical environment of collection. Analysis was completed in Stata v14.1 with collin commands used to assess collinearity. A multiple logistic regression model and a mixed effect logistic regression model were performed and compared. The significance level of p<0.05 was used.
Results: During the trial period 2,338 patients were seen. Response rates as high as 81% were seen, which was significantly improved compared with the earliest six cohorts (p<0.01). Being younger; being a new patient; having a longer wait time; having an English-speaking background and being a pre- or post-operative patient were all associated with an increased response rate of PROMs. Gender, the patient’s pathology and the type of PROM did not significantly affect the response rate.
Conclusions: The method employed to invite and inform patients of the PROM collection, and the environment in which it is undertaken, significantly alter the response rate in the completion of PROMs. Being younger, being a new patient, having a longer wait time, being English speaking, being a pre-op or post-op patient were all associated with an increase in response rate.
Keywords: patient reported outcome measures, PROM, quality of life, QOL, patient outcomes, orthopedics
Key points for decision makers
- Routine collection of patient-reported outcome measures (PROMs) is clinically important for the ongoing care of patients.
- The low response rate in the collection of PROMs is one of the major hurdles toachieving this.
- Response rate is highly variable but can be dramatically improved through implement-ing a variety of techniques.
PROMs are objective questionnaires used to measure quality of life data.1–4 Many PROMs are now considered the gold standard in measuring outcomes in orthopedics5,6 and their use improves both patient satisfaction and clinical practice.4,7 New Zealand, Sweden, and the UK have been leading the world in the collection of routine PROMs for joint registries, paving the way for quality of life data to be analyzed nationally.1,8–10 Despite the benefits associated with the collection of PROMs,11,12,21 its uptake into routine clinical practice around the rest of the world remains slow.1
There are many challenges to the routine collection of PROMs in the public hospital system1,2,10 including the time, effort and cost required.13 Although the development of a web-based repository for PROMs can help tackle some of these challenges,14,15 the utilization of this method remains limited within the orthopedic patient population.1,16,17 Another challenge is the low response rate, which is often fueled by disinterest, lack of time or inability to comprehend the questions.18 Certain characteristics that hinder the successful completion of PROMs include age greater than 75 years, social economic status, living alone, poorer pre-operative health, Hispanic or Black ethnicity, private health insurance and participants who have had previous surgeries.11,18
There is currently a lack of literature on overcoming the ongoing challenge of low response rate to patient outcome collection. The purpose of this study is to assess the predictors of completion of PROMs in the orthopedic outpatient clinic at an Australian public hospital and determine what strategies lead to an improvement in PROM response.
A prospective cohort study was conducted in an orthopedic outpatient setting of a metropolitan tertiary hospital assessing the response rate of PROM collection. Data collection occurred from February to June 2016. All patients attending the outpatient clinic were invited to complete a PROM relevant to their presenting condition via paper or digital format. All patients were asked to complete two generic wellbeing and condition-specific PROMs relevant to their presenting condition and operation (Table 1).
Table 1 Condition-specific PROMs for common operations
The primary outcome was the response rate, which was defined as fully completing and returning the PROMs back to the investigators. Data collected included patient demographics (age, sex, primary language spoken) and visit outcome. All patients were able to opt out of filling in a PROM and had their reason recorded for analysis. During the study period, eight different methods or environmental variations were employed and the response rate recorded (Table 2).
Table 2 The eight different interventions trialed throughout the study period
A bivariate analysis was performed on the demographic data (Table 3). Age and time spent were reviewed as continuous variables; sex (male vs female), language (English vs non-English speaking), PROM pathology (upper vs lower limb) and the PROM type (visual analog scales vs Likert scales) as binary variables; and visit outcome as nominal categories (pre-, post-, non-op). Continuous variables were presented as means/standard deviations, and analyzed with simple logistic regression analysis. Categorical variables were presented as absolute numbers/frequency and analyzed using a chi-squared/Fisher’s exact analysis. Those that had significance of p<0.05 were retained for model building. Collinearity was assessed between variables and one factor (PROM questionnaire) was found to be strongly correlated R ≥0.5 so removed from the multivariate analysis. A manual backward stepwise regression was used to develop a multivariate logistic regression model of predictors of response rate of PROMs. A mixed effect analysis was used to assess for any unmeasured influences by repeat observations of Cohort H over 9 weeks of intervention, and the effect applied to the other cohorts. Only factors significant at the α =0.05 level were retained in the final model, except age and sex which were considered as potential confounders. The results of the multivariate analysis (Table 4) were reported with ORs and 95% CIs. Analysis was completed in Stata v14.1 with collin commands used to assess collinearity.
Table 3 Bivariate analysis
Table 4 Multivariate analysis
Over the course of the research period, 2,338 patients were seen; of these, 414 were excluded and 1,924 patients used for analysis. Of the analyzed patients, 47.1% were males (mean age 51.16, s.d. 18.79) and 52.9% females (mean age 58.14, s.d. 17.29) (Figure 2). Gender was not found to have a significant effect on response rates between males and females (χ<0.03, p=0.87). Increasing age was found to adversely affect rate of response with a significance of (p=0.002, OR 0.99, 95% CI: 0.987, 0.997) with a significant drop in response between patients aged 60–69 compared with 50–59 (p=0.011, OR 0.69, 95% CI: 0.514, 0.918). Of the 40 different languages spoken by patients attending clinic (Table 5), English-speaking patients completed a PROM 67.18% of the time, compared with non-English speaking patients who completed it 40.05% of the time (χ2=95.78, p<0.01). New referral patients had a response rate of 68.8%, review patients with non-operative management were least likely at a rate of 44.9% and review patients who were already on a surgical waiting list had a response rate of 64.21%. The differences in response rate were found to be significant between these groups (χ=71.28, p<0.01). There was no significant difference in response rate between patients who had an upper limb pathology (61.08%) and a lower limb pathology (61.29%) (χ=0.008, p=0.93). There was no difference found in response rate between PROMs utilizing Likert scales (60.86%) and those utilizing a visual analog scale (62.73%) (χ=0.44, p=0.506). The response rate based on specific PROMs was found in patients completing the Western Ontario Shoulder Instability Index (75%), International Hip Outcome Tool (67.27%), Oxford Shoulder Score (66.67%), Manchester-Oxford Foot Score (64.39%), Oxford Hip Score (60.57%), Oxford Knee Score (59.47%), Disabilities of the Arm, Shoulder and Hand Score (58.44%) and Western Ontario Rotator Cuff Index (58.13%). However, these differences were not statistically significant (χ=11.4, p=0.122).
Table 5 List of languages spoken by patients
Figure 2 Gender age distribution.
We found that the highest response rate was in Cohort H (81.01%); this was followed by Cohort G (65.45%), Cohort E (62.79%), Cohort D (56.25%), Cohort F (54.89%), Cohort C (36.94%), Cohort B (21.11%) and Cohort A (13.33%) (Figure 1). The difference in response rate between Cohort H and the other variables were significant (χ=389, p<0.01).
There were multiple reasons why a patient did not complete a PROM. These included patients leaving the waiting room to (30%), requiring an interpreter (27%), recently completing the same PROM (19%), patients who had other physical or medical impairment not related to their orthopedic pathology that would confound the PROM results (8%), refusal (4%), missed patients (2%), vision impairment (1%), poor literacy (0.8%), incomplete PROMs (0.2%), extreme pain (0.1%), and being in handcuffs (0.1%).
There was no difference found between the naive model and the mixed effect model (Table 4). Overall the mixed-effect logistic regression model found that there was a significant difference (p<0.01) between the eight cohorts after including the independent variables into the model. Of these variables, there were multiple predictors that were found to be associated with a better response rate. These predictors were: being younger (OR=0.99, 95% CI: 0.984, 0.996); having a longer wait time (OR=1.001, 95% CI: 1.00, 1.002); being of English-speaking background (OR=3.758, 95% CI: 2.854, 4.949); being a new patient (OR=2.227, 95% CI: 1.662, 2.984) or post-operative patient (OR=1.738, 95% CI: 1.299, 2.324) compared with non-operative patients; and having the intervention performed on Cohort H compared with Cohort A (OR=0.0296, 95% CI: 0.018, 0.05), Cohort B (OR=0.0.058, 95% CI: 0.033, 0.105), Cohort C (OR=0.113, 95% CI: 0.076, 0.168), Cohort D (OR=0.334, 95% CI: 0.240, 0.465), Cohort E (OR=0.392, 95% CI: 0.251, 0.611) and Cohort F (OR=0.235, 95% CI: 0.164, 0.337).
The benefits of routine collection of PROMs are well documented.1,12,19 They improve patient satisfaction, enable early screening of functional decline and are extremely informative for clinicians.4,7,20,21 PROMs improve patient consultation and can be used for national quality control and population-based research.8,22,23 However, poor response rate continues to be a challenge that hinders its routine collection.11,18
The findings of this study show that the response rate can be significantly improved through the use of different methods or environmental variations. The most successful approach was found in Cohort H utilizing strategies including collecting PROMs prior to clinic, approaching patients individually, directing patients through an administrative area, distributing an information sheet and informing patients that the collection of PROMs was part of routine care. This approach had a response rate of 81.35%, which compares favorably with other reported PROM collection registries – UK National Joint Registry (64%), the Swedish Hip Arthroplasty Registry (79%), and the New Zealand Joint Registry (70%).8,10,24–26 We found that making PROM collection a part of routine care and the development of a specific PROM administration desk yielded the greatest improvement in response rate. We found that the response rate dropped when patients were asked to take their completed PROM into the consulting room to be assessed by the surgeon (p=0.08, OR 0.64, 95% CI 0.389, 1.058). This finding reflects the logistical difficulty of PROM collection when a multitude of users are required to participate in the collection process.
Currently, there are multiple known predictors to poor response rates described in the literature.1,11,18,27 Our study reinforces some of these predictors, including age and language.18 Other predictors such as a shorter length of time in the waiting room and review patients for non-operative treatment were also found to reduce the response rate to PROMs. Visual analog scales have been demonstrated as the most accurate measure for disabilities.25 Our study also found no difference in response rates between PROMs which utilized visual analog scales and Likert scales.
Overall the results of our study have identified factors that can be broadly split into modifiable and non-modifiable factors. The modifiable factors, such as the environmental collection modifications demonstrated in our study, were found to sequentially improve the collection response rate. Non-modifiable factors include age, language, literacy and computer skills, current stage within the surgical journey, vision impairment and patient time constraints. Measures to overcome these factors include the availability of paper format, using multi-lingual PROMs, staff availability to assist with vision impairment, literacy and computer skills, and providing information regarding the purpose of PROM collection and utilizing the patients waiting time in outpatients to collect PROMs.
This study aims to describe steps to improve the significant challenge of response rate in PROM collection. Future routine collection of PROMs will enhance the patient experience as well as for population-level quality assurance and research.2,28
There are a number of limitations that can be identified with this study. Although this was a prospective cohort study, there was no randomization. Allocation of patients into cohorts presents potential confounding variables, including temporal bias, which could be overcome by repeating each collection method over multiple clinics or randomization of patients to different methods of collection during a clinic. However, this would be logistically infeasible at our institution. Participants of this study are a small subset of orthopedic patients at a large metropolitan public hospital with particular socioeconomic, education and language variables and as such the results may not be generalized to dissimilar populations.
The future of the collection of PROMs is promising and following completion of this study, our goal is to implement the routine collection of PROMs with the lessons learned from our research. Cost and efficiency can be improved by the use of a web-based repository system to automate much of this process, requiring only the application and maintenance of the program.14 Such a system has been developed at our institution14 and successful implementation is the next major challenge that we face.
One of the major barriers to the routine collection of PROMs in the outpatient setting is the low patient response rate, and this study identifies several predictors that affect successful completion. Known predictors such as age and language were reconfirmed and others such as waiting time in clinic, new referral patients, and pre-operative patients were identified. This study also demonstrates that a high response rate can be achieved by alterations in the method of collection.
Research involving human participants and/or animals
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee. Ethics approval was obtained by the Western Health Low-Risk Human Research Ethics Panel as a quality assurance project.
A waiver of consent was sought with the approval of ethics for this project, which complied with the NHMRC (Australia) QA guidelines. This is because the collection of these data was a part of routine care and to seek consent would have been inconvenient for patients as well as potentially raising concerns/anxieties that they did not receive high-quality care. Therefore, consent was not individually sought as the data accessed for this project were being used for a purpose related to that of its original collection (ascertaining quality of care) and were to be collected by clinicians who would normally have access to that data. It would also have feasibility challenges as contacting patients may be difficult as they may have moved or changed contact details.
The authors thank the Footscray Hospital Orthopaedic Outpatient Clinic, Dr Jesse Renouf, and Dr Nicholas Pergaminelis for all of their assistance with our team in collecting the data for this study. Sincere gratitude is expressed to all of our patients who gave up their time and contributed to making our data collection successful.
All authors contributed toward data analysis, drafting and critically revising the paper, gave final approval of the version to be published, and agreed to be accountable for all aspects of the work.
The authors received no financial support for the research, authorship, and/or publication of this article, and declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
1. Black N. Patient reported outcome measures could help transform healthcare. BMJ. 2013;346:f167. doi:10.1136/bmj.f167
2. Higginson I, Carr A. Using quality of life measures in the clinical setting. Bmj. 2001;322:1297–1300. doi:10.1136/bmj.322.7297.1297
3. Antunes BB, Harding R, Higginson IJ; EUROIMPACT. Implementing patient-reported outcome measures in palliative care clinical practice: a systematic review of facilitators and barriers Lieve Vd Sarah B, Augusto C, Massimo C, Anneke F, Richard H, Irene H, Stein K, Karen L, Guido M, Bregje OP, Roeline P, Sophie P, Sheila P, Luc D KM, ed. Palliat Med. 2014;28(2):158–175. doi:10.1177/0269216313491619
4. Chow A, Mayer EK, Darzi AW, Athanasiou T, Patient-reported outcome measures: the importance of patient satisfaction in surgery. Surgery. 2009;146(3):435–443. doi:10.1016/j.surg.2009.03.019
5. Murray DW, Fitzpatrick R, Rogers K, et al. The use of the Oxford hip and knee scores. J Bone Jt Surg. 2007;89(8):1010–1014. doi:10.1302/0301-620X.89B8.19424
6. Siljander MP, McQuivey KS, Fahs AM, Galasso LA, Serdahely KJ, Karadsheh MS. Current trends in patient-reported outcome measures in total joint arthroplasty: a study of 4 major orthopaedic journals. J Arthroplasty. 2018;33:3416–3421. doi:10.1016/j.arth.2018.06.034
7. Chen J, Ou L, Hollis SJ. A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Serv Res. 2013;13:211. doi:10.1186/1472-6963-13-438
8. Rolfson O, Karrholm J, Dahlberg LE, Garellick G. Patient-reported outcomes in the Swedish hip arthroplasty register: results of a nationwide prospective observational study. Bone Joint J. 2011;93-B(7):867–875. doi:10.1302/0301-620X.93B7.25737
9. Green M, Wishart N, Young E, Mccormack V, Swanson M. 13th Annual Report; 2016. National Joint Registry for England, Wales, Northern Ireland and the Isle of Man. Available from: http://www.njrcentre.org.uk/njrcentre/Portals/0/Documents/England/Reports/13th%20Annual%20Report/07950%20NJR%20Annual%20Report%202016%20ONLINE%20REPORT.pdf. Accessed May 7, 2019.
10. Rothwell A, Larmer P, Hobbs T, Rothwell A. The New Zealand Joint Registry; sixteen year report; January 1999 to December 2014. Available from: https://nzoa.org.nz/system/files/Web_DH7657_NZJR2014Report_v4_12Nov15.pdf. Accessed May 7, 2019.
11. Hutchings A, Neuburger J, Grosse Frie K, Black N, van der Meulen J. Factors associated with non-response in routine use of patient reported outcome measures after elective surgery in England. Health Qual Life Outcomes. 2012;10:34. doi:10.1186/1477-7525-10-34
12. Davis JC, Bryan S, Patient Reported Outcome Measures (PROMs) have arrived in sports and exercise medicine: why do they matter? Br J Sports Med. 2015;49(24):1545–1546. doi:10.1136/bjsports-2014-093707
13. Falavigna A, Dozza DC, Teles AR, et al. Current status of worldwide use of patient-reported outcome measures (PROMs) in spine care. World Neurosurg. 2017;108:328–335. doi:10.1016/j.wneu.2017.09.002
14. Tirosh O, Tran P, Renouf J, et al. PROMsBase: web-based repository portal for patient-reported outcome measures in orthopaedics. Health Informatics J. 2017:1460458217725904. doi:10.1177/1460458217725904
15. Wu AW, Kharrazi H, Boulware LE, Snyder CF, Measure once, cut twice–adding patient-reported outcome measures to the electronic health record for comparative effectiveness research. J Clin Epidemiol. 2013;66(8Suppl):S12–S20. doi:10.1016/j.jclinepi.2013.04.005
16. Fraval A, Chong YM, Holcdorf D, Plunkett V, Tran P. Internet use by orthopaedic outpatients - Current trends and practices. Australas Med J. 2012;5(12):633–638. doi:10.4066/AMJ.2012.1530
17. Palmen LN, Schrier JCM, Scholten R, Jansen JHW, Koeter S, Is it too early to move to full electronic PROM data collection?: A randomized controlled trial comparing PROM’s after hallux valgus captured by e-mail, traditional mail and telephone. Foot Ankle Surg. 2016;22(1):46–49. doi:10.1016/j.fas.2015.05.001
18. Schamber EM, Takemoto SK, Chenok KE, Bozic KJ, Barriers to completion of patient reported outcome measures. J Arthroplasty. 2013;28(9):1449–1453. doi:10.1016/j.arth.2013.06.025
19. Weldring T, Smith SMS. Patient-Reported Outcomes (PROs) and Patient-Reported Outcome Measures (PROMs). Heal Serv Insights. 2013;6:61–68.
20. Greenhalgh J, Meadows K. The effectiveness of the use of patient-based measures of health in routine practice in improving the process and outcomes of patient care: a literature review. J Eval Clin Pract. 1999;5(4):401–416.
21. Skevington SM, Day R, Chisholm A, Trueman P. How much do doctors use quality of life information in primary care? Testing the trans-theoretical model of behaviour change. Qual Life Res. 2005;14(4):911–922.
22. Stover AM, Basch EM, Using patient-reported outcome measures as quality indicators in routine cancer care. Cancer. 2016;122(3):355–357. doi:10.1002/cncr.29768
23. Marshall S, Haywood K, Fitzpatrick R, Impact of patient-reported outcome measures on routine practice: a structured review. J Eval Clin Pract. 2006;12(5):559–568. doi:10.1111/j.1365-2753.2006.00650.x
24. National Joint Registry. 12th annual report. 12th Annu Rep. 2015;12(December2014):81–88.
25. Boonstra AM, Schiphorst Preuper HR, Reneman MF, Posthumus JB, Stewart RE. Reliability and validity of the visual analogue scale for disability in patients with chronic musculoskeletal pain. Int J Rehabil Res. 2008;31(2):165–169. doi:10.1097/MRR.0b013e3282fc0f93
26. Prodinger B, Taylor P. Improving quality of care through patient-reported outcome measures (PROMs): expert interviews using the NHS PROMs programme and the Swedish quality registers for knee and hip arthroplasty as examples. BMC Health Serv Res. 2018;18(1):87. doi:10.1186/s12913-018-2898-z
27. Jahagirdar D, Kroll T, Ritchie K, Wyke S. Using patient reported outcome measures in health services: a qualitative study on including people with low literacy skills and learning disabilities. BMC Health Serv Res. 2012;12:431. doi:10.1186/1472-6963-12-431
28. Dawson J, Doll H, Fitzpatrick R, Jenkinson C, Carr AJ. The routine use of patient reported outcome measures in healthcare settings. BMJ. 2010;340:c186. doi:10.1136/bmj.c293
29. Hudak PL, Amadio PC, Bombardier C. Development of an upper extremity outcome measure: the DASH (disabilities of the arm, shoulder and hand) [corrected]. The Upper Extremity Collaborative Group (UECG). Am J Ind Med. 1996;29(6):602–608. doi:10.1002/(SICI)1097-0274(199606)29:6<602::AID-AJIM4>3.0.CO;2-L
30. Dawson J, Rogers K, Fitzpatrick R, Carr A. The Oxford shoulder score revisited. Arch Orthop Trauma Surg. 2009;129(1):119–123. doi:10.1007/s00402-007-0549-7
31. Mohtadi NGH, Griffin DR, Pedersen ME, et al. The Development and validation of a self-administered quality-of-life outcome measure for young, active patients with symptomatic hip disease: the international Hip Outcome Tool (iHOT-33). Arthroscopy. 2012;28(5):595–605. quiz 606-10.e1. doi:10.1016/j.arthro.2012.03.013
32. Wessel RN, Lim TE, Van Mameren H, De Bie RA. Validation of the western ontario rotator cuff index in patients with arthroscopic rotator cuff repair : A study protocol. BMC Musculoskelet Disord. 2011;12(1):64. doi:10.1186/1471-2474-12-64
33. Kirkley A, Griffin S, McLintock H, Ng L. The development and evaluation of a disease-specific quality of life measurement tool for shoulder instability. The Western Ontario Shoulder Instability Index (WOSI). Am J Sports Med. 1998;26(6):764–772. doi:10.1177/03635465980260060501
34. Morley D, Jenkinson C, Doll H, et al. The Manchester – Oxford Foot Questionnaire (MOXFQ) development and validation of a summary index score. Bone Joint Res. 2013;2(4):66–69. doi:10.1302/2046-3758.24.2000147
35. Hawker GA, Mian S, Kendzerska T, French M. Measures of adult pain: Visual Analog Scale for Pain (VAS Pain), Numeric Rating Scale for Pain (NRS Pain), McGill Pain Questionnaire (MPQ), Short-Form McGill Pain Questionnaire (SF-MPQ), Chronic Pain Grade Scale (CPGS), Short Form-36 Bodily Pain Scale (SF). Arthritis Care Res. 2011;63(Suppl.11):240–252. doi:10.1002/acr.20543
36. Devlin N, Shah K, Feng Y. Valuing health-related quality of life : an EQ-5D-5L value set for England. Health Econ. 2018;27(1):7-22. doi:10.1002/hec.3564.
This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.Download Article [PDF]