Subjects with osteoarthritis can easily use a handheld touch screen electronic device to report medication use: qualitative results from a usability study

Introduction

Osteoarthritis (OA) is a chronic joint disease and the most frequent cause of disability among adults in the United States. OA most commonly affects joints of the knee, hip, hands, and spine.¹ Data from the 2010 to 2012 National Health Interview Survey determined that >52 million adults in the United States had at least one type of clinically diagnosed OA.² By 2030, this number is estimated to reach 67 million, or 25% of Americans older than 18 years.³ This is attributed to increases in aging and obese populations.^4,5 The total cost associated with arthritis care in the United States in 2003 was $128 billion; two-thirds of this went to medical costs, while the remaining third was due to earning losses.^6,7 OA was the fourth most common cause of hospitalization in 2009.⁷

OA clinical care is largely focused on alleviating symptoms and improving health-related quality of life as assessed by patients.⁸ Selecting an appropriate outcome mode is critical to the design of clinical trials and patient care. Patient-reported outcomes (PROs) are becoming an increasingly popular and reliable method for collecting patient health and quality-of-life data.⁹ A study that examined 215 prescription drugs approved in the United States from 1997 to 2002 found that 30% of the product labels utilized PRO endpoints.¹⁰ In addition, PROs were the only outcome in 11% of these US Food and Drug Administration-approved product labels. PROs are especially valuable when the ultimate treatment goal is to alleviate pain and to improve quality of life¹¹ and physical functioning, as is the case with OA therapy. Pain and discomfort are predominant symptoms of OA, and therefore, subjective patient assessment is essential for OA diagnosis and care.¹² In addition, there are other elements to the pain in OA, which may have further treatment implications; as the disease progresses, the pain experienced may become chronic, unexpected, or even indicate concomitant disorders, such as neuropathy.^13,14 Currently, the National Health Service in England requires patients to be monitored using PROs before and after certain surgeries, including knee and hip replacement.^15,16

In 1997, the Outcome Measures in Rheumatoid Arthritis Clinical Trials (OMERACT) committee identified a core set of outcome assessments for OA clinical trials that require patients to self-report on pain, physical function, and global assessments.¹⁷ More recently, the Osteoarthritis Research Society (OARSI), together with OMERACT, proposed a simplified set of patient-reported criteria to establish international standardization across OA clinical trials and to include assessment of symptoms prior to patients entering OA clinical trials.¹⁸ This is essential as the success of OA clinical trials is determined by a measurable improvement in symptoms. To this end, it is also necessary to ensure that changes in pain level are detectable from data provided by PROs. Tubach et al set out to establish the minimal clinically important improvement reported by patients with knee or hip OA using the OMERACT–OARSI outcome measures: pain, physical function, and patient global assessment.¹³ They confirmed minimal clinically important improvement as one of the useful clinical criteria to measure OA symptom improvement based on individual patient opinion and perspective. A number of PRO instruments that meet OARSI measure guidelines have been recommended for use in OA clinical trials (reviewed in Fitzgerald et al¹⁹). In addition, feasibility testing on several PRO instruments, such as EQ-5D, SF-36, HOOS, and OHS, demonstrated that they are acceptable to and appropriate for use by OA patients.²⁰ Furthermore, the Hip/Knee Osteoarthritis Decision Quality Instrument was found feasible and acceptable to facilitate high-quality decisions for patients considering joint replacement surgery.²¹

With the success and validation of PROs for clinical trial endpoints, particularly for OA, comes an increasing use of electronic methods to capture patient data. Electronic PROs (ePROs) are recognized by the US Food and Drug Administration as a promising clinical trial outcome measure.²² ePROs offer many advantages over paper and pencil instruments, including less time to completion and decreased patient burden, elimination of missing data points and data entry mistakes, increased patient compliance, and improved accuracy of data collection.^23–25 Previous studies recorded response rates of only 72%–83.4% for paper administration of PROs to patients with OA.^20,26,27 In addition, one study reported a rate of missing response items up to 4.3%.²⁰ Implementation of ePROs for this patient population may increase compliance and eliminate missing responses.

Despite many advantages to paper administration, it is necessary to provide sufficient evidence for measurement equivalence of ePRO devices, including cognitive debriefing, equivalency testing, and usability testing.²⁸ A recent review of PRO and ePRO equivalency studies, including general and condition-specific instruments, found that 78% of studies demonstrated equivalence between paper and electronic formats.²⁹ In addition, 87% of publications that included preference data found that patients preferred electronic methods over paper administration. As PROs are especially useful for therapeutic areas, such as OA, it is essential to collect ePRO equivalency and preference data for these patients specifically. A web-based ePRO system demonstrated excellent equivalency to pencil and paper administrations of the OHS, McCarthy hip score, UCLA activity score, and howRU score on OA patients.³⁰ Furthermore, several additional instruments were tested on electronic handheld devices by subjects with rheumatoid arthritis.³¹ All seven of the surveys tested were found to be equivalent between modes of administration. Importantly, subjects found that the electronic method is easy to use and preferred this mode over paper, regardless of age or previous experience with technology.

ePRO is a promising method for capturing patient data, particularly in a chronic condition such as OA, where pain symptoms and quality of life are best assessed by the patients. Although mode equivalency and preference studies have demonstrated appropriate use of ePROs for patients with OA, there is very little information on usability testing for this group. Especially in the case of OA, usability studies should consider potential physical pain or discomfort when using devices. The objective of this study was to conduct usability testing on a General Symptom Questionnaire and a Medication Module™ using the LogPad^® LW handheld device for subjects with OA.

Methods

The usability testing described in this report was conducted in the context of a larger randomized, crossover mode equivalence study measuring the equivalence of a standardized PRO questionnaire when administered on paper and on two electronic modalities (N=400). Subjects who participated in this usability testing (n=9) did so immediately after participating in the equivalence study. Some data collected as part of the equivalence study (eg, demographics and technology use) were used in the analysis for the usability testing. The purpose of the usability testing was to determine whether subjects had any difficulty in using the LogPad^® LW device or understanding the General Symptom Questionnaire or the Medication Module™ as they were presented on the LogPad^® LW handheld electronic diary.

Ethics

This study was reviewed and approved by Copernicus Group independent review board (IRB) prior to any contact with study subjects. The following study documents were submitted to the IRB: protocol, informed consent form, demographic and condition questionnaire, and interview guide (IG). Following review, IRB approval was granted on April 28, 2014.

Questionnaires

A General Symptom Questionnaire was created that consists of three screens: 1) an instruction screen, 2) a single-select question about subject fatigue, and 3) an alert screen. The answer options on screen 2 were one-word descriptors on a four-point scale of intensity (none, mild, moderate, and severe). As the subject selected one of the answer options, a more detailed definition of that answer dynamically displayed at the bottom of the screen, and subjects could change their selection to view the full definition of each answer option. The alert screen is an example of the type of message that might be displayed to the patient if their reported symptoms are worsening over time.

The Medication Module™ is a component of the eResearch Technologies, Inc (ERT) ePRO system used to track subject’s consumption of medications. It enables entry of medication data by subjects via the ERT LogPad^® LW handheld device and allows subjects to record medications between visits and identify when any new medications have been taken that have not previously been described. The Medication Module™ consists of several sections. First, the subject indicates whether or not they took any medications since yesterday. If they select “No”, they are directed to the end of the questionnaire; if they select “Yes”, they are directed to a screen which displays commonly used drug/dose/mode combinations that are tailored to the subject. The subject selects one of these medications or “Other”, then enters the date, time, and dose amount of the medication. Subjects can then choose to add another time/dose of the same medication, add a different medication, or review medications they have already entered. Once they finish adding medications, they are shown a review screen displaying all medications they have added. Subjects can edit or delete any of these medications or continue to add new medications until the review screen accurately displays the medications they have taken that day.

Recruitment

Following IRB approval, subjects were recruited for the main PRO equivalence study. Advertisements were placed in local media in Boston, MA, and potentially eligible subjects contacted ERT via phone or email. ERT personnel spoke with subjects via phone and screened them for eligibility; eligible subjects were scheduled an appointment at the study site in Boston, MA. Subjects arrived at the site and participated in the main PRO equivalence study; after completing the study, ERT personnel approached subjects about participating in an additional 15-minute usability test and interview. Consecutive subjects were approached until the target sample size for usability testing was achieved.

Sample size

Previous research suggests that a minimum of 3 and a maximum of 20 subjects is an appropriate sample size for a qualitative usability study, depending on the complexity of the study.^32,33 The general consensus is that five participants will uncover 80% of the usability issues that should arise^34,35 and seven or more participants would be an optimal number.³⁶ Recruitment for this usability study targeted nine participants.

Eligibility criteria

Subjects recruited for the usability study had already met the eligibility criteria of the main PRO equivalence study. Eligible subjects for the equivalence study were 18 years or older, provided a signed informed consent form indicating an understanding of the study objectives and study procedures and a willingness to participate in the study, had the ability to complete surveys alone or without assistance in English, reported having been told by a doctor or other health professional that they had OA or degenerative arthritis, and reported having pain, swelling, or limitation in range of motion in a knee or hip during the past 4 weeks. Subjects were ineligible if they reported having been medically diagnosed with traumatic brain injury, dementia, schizophrenia, any psychoses with current symptoms, or alcohol or drug dependence with current symptoms. The only additional eligibility criterion for the usability study was that subjects verbally consent to participate in the study.

Interviews

Prior to conducting any interviews, the study research team created a semistructured IG to identify potentially challenging aspects of the interview and highlight the most important questions to address in order to answer the research questions. Subjects who consented to participate were interviewed by ERT personnel who have undergone National Institutes of Health Human Participant Protection training, as well as training for subject interviews. The interviews were conducted in a private room to ensure confidentiality.

The interviewer introduced him/herself and gave an overview of the objectives of the interview. The interviewer then turned on the audio recorder and secured additional verbal consent from the subject to have the interview audio recorded. The subject was then given a LogPad^® LW device with the General Symptom Questionnaire and the Medication Module™ and asked to complete each questionnaire as if they were participating in a clinical trial.

Following completion of both questionnaires, the interview began. A semistructured IG was used to obtain both structured and spontaneous feedback from subjects. The guide included open-ended questions to collect qualitative data from the subject’s perspective and was constructed to establish the usability of both the LogPad^® LW itself and the questionnaires implemented on it. After completion of the interview, each subject was compensated with a $100 gift card.

Qualitative data analysis

Audio recordings of the interviews were transcribed verbatim and anonymized by removing identifying information, such as names. Transcripts were manually coded in Microsoft Excel^® by one research analyst, using predetermined codes that aligned with questions in the IG and expected subject response categories. Subject responses that were not adequately described by a predetermined code were categorized with new codes and direct subject quotes were extracted from the interview transcript as supplementary data. Another research analyst reviewed and aggregated the coded data from all transcripts. Each transcript was considered a unit of analysis; a third researcher performed a descriptive analysis on the aggregated data.

Results

Socioeconomic demographics, comorbid conditions, and technology use

Information on socioeconomic demographics and comorbid conditions reported by each subject on the demographic and conditions questionnaire is presented in Table 1. Information on technology use and preferences reported by each subject on the demographic and conditions questionnaire is presented in Table 2. In addition to these data collected on paper, subjects were asked during the interview to describe their level of familiarity and confidence with using touch screen devices using a four-point rating scale. These responses are presented in Table 3.

Table 1 Demographic and conditions information (N=9) Abbreviations: GED, General Educational Development; AA, Associate of Arts.

Table 2 General technology use information (N=9)

Table 3 Familiarity and confidence with touch screen devices (N=9)

Overall, subjects were middle aged or older with an annual household income of $20,000/year or less; none had a college degree. Three subjects reported rheumatoid arthritis as a comorbid condition. Current level of technology use varied widely, with about half of subjects reporting that they never used Internet or email and others using Internet and email daily. Self-reported familiarity and confidence with touch screen devices was low, with no subjects reporting that they were “very familiar” or “very confident” using these devices.

Usability testing of the General Symptom Questionnaire and ERT Medication Module™ with subjects with OA

Following completion of the General Symptom Questionnaire, subjects were asked a series of questions regarding the usability of this questionnaire on the device. Data from these questions are shown in Table 4, with selected quotes from subject interviews in italics. Subjects also completed the Medication Module™ to track their pain medication usage. After finishing the Medication Module™, participants were asked a series of questions regarding usability of this medication diary on the device. Data from these questions are summarized in Table 5. Overall, subjects did not encounter any significant issues when completing the General Symptom Questionnaire or Medication Module™.

Table 4 Usability testing of the General Symptom Questionnaire on LogPad® LW (N=9)

Table 5 Usability performance of the ERT Medication Module™ on LogPad® LW (N=9) Abbreviation: ERT, eResearch Technologies Inc.

Usability testing of the LogPad^® LW with subjects with OA

We asked subjects several questions about the comfort and portability of the LogPad^® LW device; data from these questions are presented in Table 6. Overall, subjects did not experience any physical discomfort during completion of either questionnaire and were satisfied with the size and portability of the device. The majority of subjects were willing to use the LogPad^® LW handheld device to complete diaries at home or in public.

Table 6 Usability testing of the LogPad® LW (N=9)

Discussion

Regulatory guidelines and scientific evidence indicate that there is improved data quality with electronic data capture. As such, there is increased use of ePROs to improve the collection and quality of PRO data for clinical trials, postmarketing studies, disease management studies, and clinical care. In this study, we tested usability of a General Symptom Questionnaire and Medication Module™ on the LogPad^® LW handheld device. The General Symptom Questionnaire was created to record patient symptoms using standard types of response scales and question/response formats, while the Medication Module™ provides patients with a way to record their pain medications using the device.

During the course of the usability testing, some subjects alluded to the fact that they were using the device for the first time and might have questions. This is a common concern among subjects, and for this reason, we suggest that when ePRO assessments are deployed in clinical trials, subjects be provided custom training and complete a practice diary on site with clinical staff present before they are required to complete any diaries at home by themselves. This provides the opportunity for subjects to ask questions and become acquainted with the functionality of the device, which can increase compliance, data quality, and patient engagement in the trial.

Conclusion

In summary, subjects did not encounter any significant issues in completing the General Symptom Questionnaire or the Medication Module™ on the handheld device. Despite reporting limited daily technology experience and limited confidence and familiarity with touch screen devices, all nine subjects found the device screens easy to use, found it easy to progress through the questionnaires, and thought that they could accurately answer items using the device. Although this usability testing was conducted using subjects with OA (three of which reported also having rheumatoid arthritis), none of the subjects reported experiencing physical discomfort when using the handheld device. These findings are consistent with previously published studies demonstrating the usability of electronic data capture of PROs in patients with OA and other rheumatologic conditions.^37–39

Limitations of this study include the possibility of selection bias. We did not document the number of subjects who declined to participate in the usability testing and are thus unable to compare their demographic characteristics with the subjects in our study to determine whether any selection bias occurred. A second limitation is around the generalizability of our results. We used a convenience sample of nine subjects, thus it is possible that the demographics, technology experience, or other characteristics of the subjects do not exactly reflect the characteristics of the average OA patient. For example, about half of all US people older than 65 years have been diagnosed with OA,² while our sample contained only two subjects (22%) older than 65 years. Furthermore, usability research using a larger, more representative sample of subjects may provide firmer empirical evidence of generalizability; however, we are not convinced that it will bring to light any additional significant usability findings.

Funding

This study was supported by ERT.

Disclosure

The authors are employees of ERT. The authors report no other conflicts of interest in this work.

---

References

1.		Brandt KD, Dieppe P, Radin E. Etiopathogenesis of osteoarthritis. Med Clin North Am. 2009;93:1–24.
2.		Barbour KE, Helmick CG, Theis KA, et al. Prevalence of doctor-diagnosed arthritis and arthritis-attributable activity limitation – United States, 2010–2012. MMWR Morb Mortal Wkly Rep. 2013;62:2010–2012.
3.		Hootman JM, Helmick CG. Projections of US prevalence of arthritis and associated activity limitations. Arthritis Rheum. 2006;54:226–229.
4.		Wendelboe AM, Hegmann KT, Biggs JJ, et al. Relationships between body mass indices and surgical replacements of knee and hip joints. Am J Prev Med. 2003;25:290–295.
5.		Karlson EW, Mandl LA, Aweh GN, Sangha O, Liang MH, Grodstein F. Total hip replacement due to osteoarthritis: the importance of age, obesity, and other modifiable risk factors. Am J Med. 2003;114:93–98.
6.		Yelin E, Murphy L, Cisternas MG, Foreman AJ, Pasta DJ, Helmick CG. Medical care expenditures and earnings losses among persons with arthritis and other rheumatic conditions in 2003, and comparisons with 1997. Arthritis Rheum. 2007;56:1397–1407.
7.		Murphy L, Helmick CG. The impact of osteoarthritis in the United States. Orthop Nurs. 2012;31:85–91.
8.		Conaghan PG, Dickson J, Grant RL. Care and management of osteoarthritis in adults: summary of NICE guidance. BMJ. 2008;336:502–503.
9.		Bottomley A, Jones D, Claassens L. Patient-reported outcomes: assessment and current perspectives of the guidelines of the Food and Drug Administration and the reflection paper of the European Medicines Agency. Eur J Cancer. 2008;45:347–353.
10.		Willke RJ, Burke LB, Erickson P. Measuring treatment impact: a review of patient-reported outcomes and other efficacy endpoints in approved product labels. Control Clin Trials. 2004;25:535–552.
11.		Deshpande PR, Rajan S, Sudeepthi BL, Abdul Nazir CP. Patient-reported outcomes: a new era in clinical research. Perspect Clin Res. 2011;2:137.
12.		Bijlsma JWJ, Berenbaum F, Lafeber FPJG. Osteoarthritis: an update with relevance for clinical practice. Lancet. 2011;377:2115–2126.
13.		Tubach F, Ravaud P, Baron G, et al. Evaluation of clinically relevant changes in patient reported outcomes in knee and hip osteoarthritis: the minimal clinically important improvement. Ann Rheum Dis. 2005;64:29–33.
14.		Hawker GA, Stewart L, French MR, et al. Understanding the pain experience in hip and knee osteoarthritis – an OARSI/OMERACT initiative. Osteoarthritis Cartilage. 2008;16:415–422.
15.		Devlin NJ, Parkin D, Browne J. Patient-reported outcome measures in the NHS: new methods for analysing and reporting EQ-5D data. Health Econ. 2010;905:886–905.
16.		Department of Health. Guidance on the Routine Collection of Patient Reported Outcome Measures (PROMs). 2009:1–28. Available from: http://webarchive.nationalarchives.gov.uk/+/www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_092647. Accessed May 25, 2016.
17.		Bellamy N, Kirwan J, Boers M, et al. Recommendations for a core set of outcome measures for future phase III clinical trials in knee, hip, and hand osteoarthritis. Consensus development at OMERACT III. J Rheumatol. 1997;24:799–802.
18.		Pham T, Der Heijde D, Van Lassere M, et al. Outcome variables for osteoarthritis clinical trials: the OMERACT-OARSI set of responder criteria. J Rheumatol. 2003;30:1648–1654.
19.		Fitzgerald GK, Hinman RS, Zeni J, et al. OARSI clinical trials recommendations: design and conduct of clinical trials of rehabilitation interventions for osteoarthritis. Osteoarthritis Cartilage. 2015;23:803–814.
20.		Paulsen A, Pedersen AB, Overgaard S, Roos EM. Feasibility of 4 patient-reported outcome measures in a registry setting. Acta Orthop. 2012;83:321–327.
21.		Sepucha KR, Stacey D, Clay CF, et al. Decision quality instrument for treatment of hip and knee osteoarthritis: a psychometric evaluation. BMC Musculoskelet Disord. 2011;12:149.
22.		Patrick DL, Burke LB, Powers JH, et al. Patient-reported outcomes to support medical product labeling claims: FDA perspective. Value Health. 2007;10(suppl 2):S125–S137.
23.		Broderick JE. Electronic diaries: appraisal and current status. Pharmaceut Med. 2008;22(2):69–74.
24.		Stone AA, Shiffman S, Schwartz JE, Broderick JE, Hufford MR. Patient compliance with paper and electronic diaries. Control Clin Trials. 2003;24:182–199.
25.		Dale O, Hagen KB. Despite technical problems personal digital assistants outperform pen and paper when collecting patient diary data. J Clin Epidemiol. 2007;60:8–17.
26.		Wylde V, Blom AW, Whitehouse SL, Taylor AH, Pattison GT, Bannister GC. Patient-reported outcomes after total hip and knee arthroplasty. Comparison of midterm results. J Arthroplasty. 2009;24:210–216.
27.		Escobar A, Quintana JM, Bilbao A, et al. Effect of patient characteristics on reported outcomes after total knee replacement. Rheumatology (Oxford). 2007;46:112–119.
28.		Coons SJ, Gwaltney CJ, Hays RD, et al. Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO good research practices task force report. Value Health. 2009;12:419–429.
29.		Campbell N, Ali F, Finlay AY, Salek SS. Equivalence of electronic and paper-based patient-reported outcome measures. Qual Life Res. 2015;24(8):1949–1961.
30.		Griffiths-Jones W, Norton MR, Fern ED, Williams DH. The equivalence of remote electronic and paper patient reported outcome (PRO) collection. J Arthroplasty. 2014;29:2136–2139.
31.		Tiplady B, Goodman K, Cummings G, et al. Patient-reported outcomes in rheumatoid arthritis. Patient. 2010;3:133–143.
32.		Virzi RA. Refining the test phase of usability evaluation: how many subjects is enough? Hum Factors. 1992;34:457–468.
33.		Perfetti C, Landesman L [webpage on the Internet]. Eight Is Not Enough. 2001. Available from: http://www.uie.com/articles/eight_is_not_enough. Accessed July 15, 2015.
34.		Nielsen J, Landauer TK. A mathematical model of the finding of usability problems. Proceedings of: ACM INTERCHI’93 Conference. Amsterdam: ACM; 1993:206–213.
35.		Nielsen [homepage on the Internet]. Why You Only Need to Test with 5 Users. 2000. Available from: www.useit.com/alertbox/20000319.html. Accessed February 22, 2009.
36.		Turner CW, Lewis JR, Nielsen J. Determining usability test sample size. In: Karwowski W, Raton B, editors. International Encyclopedia of Ergonomics and Human Factors. Vol. 3. Second ed. Boca Raton, FL: CRC Press; 2006:3084–3088.
37.		Bellamy N, Patel B, Davis T, et al. Electronic data capture using the womac NRS 3.1 index (m-Womac): a pilot study of repeated independent remote data capture in OA. Inflammopharmacology. 2010;18(3):107–111.
38.		Lee SJ, Kavanaugh A, Lenert L. Electronic and computer-generated patient questionnaires in standard care. Best Pract Res Clin Rheumatol. 2007;21(4):637–647.
39.		Salaffi F, Gasparini S, Ciapetti A, Gutierrez M, Grassi W. Usability of an innovative and interactive electronic system for collection of patient-reported data in axial spondyloarthritis: comparison with the traditional paper-administered format. Rheumatology (Oxford). 2013;52(11):2062–2070.