Patient Perspective in the Development of Electronic Patient-Reported Outcomes (ePROs) in Seizure Disorders: A Patient-Centric Approach

Introduction

Patient-centered outcomes research (PCOR) improves the quality of care and patient outcomes by utilizing patient input to inform the research process and development of assessment tools. Given PCOR’s emphasis on the patient perspective, methods to incorporate patient-reported outcomes (PROs) are important. Patient-reported outcomes (PRO) could be used at a group level and at the individual patient level to assist with decision-making by clinicians and researchers. The US Food and Drug Administration describes a patient-reported outcome (PRO) as an assessment hinged on “any report of the status of a patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or anyone else”¹ This definition highlights a general, patient-oriented viewpoint with a methodical feature. As noted by Schwartzberg, in the evolving team-based methodology to delivering medical care, gathering PROs affords longitudinal monitoring of adverse effects of treatment, barriers or complications of the disease, functional statuses, and psychological conditions throughout the course of the disease for all stakeholders to use.² To ensure the creation of measurement tools that maximize value at the lowest cost, all aspects of the health care system need to align with patient needs and preferences from scale development to electronic PRO development.

At the population level, clinical information is collected using health surveys, disease registries, and administrative claims. Collection of PROs at the population level produces various opportunities to incorporate PROs in PCOR. For example, PROs are commonly used in epilepsy and seizure disorders to facilitate clinical diagnosis and evaluate treatment efficacy. Epilepsy is one of the most common neurological diseases with a prevalence ranging from 5 to 9 per 1000 persons.³ Studies concentrating on seizure-free and nonseizure-free patients have shown that seizure freedom is the most influential factor involved in a satisfactory quality of life.⁴ There are a number of symptoms and impacts of seizures which cannot be observed by clinicians and may be more reliably reported by patients and/or caregivers themselves. Such information is increasingly obtained through electronic assessment, which captures symptom features in the absence of direct observation by a clinician.⁵ Prior studies have shown that electronic and paper-and-pencil PROs delivered equivalent measures.^6,7 Additionally, the use of ePROs is associated with several advantages over pen-and-paper methods, particularly in terms of reducing missing data, providing time-stamped records, minimizing administrative burden, and avoiding secondary data entry errors.⁵ Therefore, it is crucial to demonstrate the “usability” of the ePRO or that patients can use the software and the device appropriately.

Although ePROs have been implemented successfully in many populations, there is insufficient evidence on patient perceptions on scale items and ePRO performance in epilepsy and seizure disorders. Recent technological developments facilitate the electronic collection of PROs and linkage of PRO data, offering new opportunities for putting the patient perspective in PCOR. Seizures are not timed events. Knowing that patients are under no direct obligation to enter data on a real-time basis, and that they can wait days or even weeks and then try to recall their experience, ePRO can provide the necessary assurance that data were entered by the patient at the time the episode occurs. Developing an ePRO should start with a comprehensive patient engagement platform, with a complete review of systems survey and capabilities for mobile health usage. The aim of the present study was to provide a wide-ranging assessment of patient perceptions of completing ePROs, expectations of ePRO devices for PCOR and on-site clinical visit in order to guide the development of successful ePRO deployment in seizure-related disorders.

Methods

Participants

Participants were included in the study if their primary language was English, and were adult male or female >18 to 85 years old. Exclusion criteria included, non-English speaking <18 years old or >85 years old, and history of traumatic brain injury or severe neurological deficit that may make it difficult to complete an online questionnaire.

We determined this study is exempt from IRB review because it does not meet the definition of human subject as defined in 45 CFR 46.102. Specifically, this research involves analysis of anonymous data obtained from a previously conducted online market research survey of patients about their experiences with patient-reported outcome measures. The investigator analyzed the anonymous responses. The investigator is not able to identify the individuals who provided the responses based on any linked identifiers. Therefore, the individuals involved are not considered human subjects based on the definition in the regulations. The study was submitted to Western IRB and this entity determined that it was exempt from IRB approval.

A total of 713 individual responses were completed via the online survey from 15 March 2019 to 30 April 2019. The survey was developed using Qualtrics (https://www.qualtrics.com) and distributed to individuals via email through clinicalconnection.com. Individuals were compensated for completion of the survey with entry into a drawing for a gift card. Of the 713 individuals, 19 were removed due to duplicate records obtained from the same IP address with identical demographic data, and an additional 54 were removed, as they did not respond to any of the situational questions (SQs), for a total of 640 participants.

Situational Questions (SQs)

First, a panel of experts was established consisting of scientists, clinicians with experience in seizures, and Electronic Clinical Outcome Assessment (eCOA) technologists to develop key questions and conduct literature searches to identify evidence on which to develop recommendations for questions and responses. Literature searches were conducted for articles published from 2009 to 2019. All relevant identified English-language articles were included for review. No studies of sufficient quality were identified upon which to make evidence-based recommendations. This deductive method, also known as “logical partitioning” or “classification from above” is based on the description of the relevant domain and the identification of items.⁸

Next, we considered the content to be included in the development of the technology survey to assess insights into the development of ePROs for PCOR. The content was developed utilizing a panel of experts consisting of scientists and Electronic Clinical Outcome Assessment (eCOA) technologists were invited to participate. Based on the literature, we identified the procedures, risks, benefits, and alternatives for development of a survey to assess individuals’ perceptions and recommendations for questions related to seizures, and development of technological platforms for seizures. The inductive method, also known as “grouping” involved the generation of items from the responses of these scientists and eCOA technologists. Qualitative data obtained through focus groups and individual interviews with clinicians were used to inductively identify items related to technology and seizure reporting. Studies have shown that too much information to the patients can become overwhelming;⁹ therefore, the content of the questionnaire should contain the precise information that participants wish to discern within ethical principles and legal regulations.

The panel first generated theoretically derived items and then subjected them to a content validity assessment by eight eCOA technologists and scientists who were asked to classify each randomly ordered item to categories of relevancy to eCOA or not relevant to eCOA development. Those items that were assigned to the relevancy to eCOA development category more than 80% of the time were retained for use in the questionnaire. The 11 items. The questionnaire related to seizure items resulted in 11 items (see Table 1). It was determined that Situational Questions (SQs) would be used as they are similar to behavioral questions. However, instead of asking the participant to relay a past experience and tell what they thought of it in that situation, they are presented with a hypothetical situation.

Table 1 Distribution of Questions Developed for the Survey

All participants were asked to complete a survey consisting of demographic questions (age, gender, highest level of education; current psychiatric or medical diagnosis) and situational interview questions (SQs; see Table 1). In many cases, SQs involve problem-solving and handling issues and circumstances when using technological devices or developing content for PROs or ePROs. Participants were asked to respond by selecting the most appropriate response from a pre-determined list of responses.

Data Processing and Statistical Analysis

The data collected from the participants were identified only by a participant number without any information specific to the patient to protect the patients’ privacy and secure confidentiality. Descriptive statistics were performed to analyze the baseline characteristics of the participants. The means and standard deviations were calculated for normally distributed continuous variables, and proportions were calculated for categorical variables. The differences in the participants’ ratings of each item between age category (<30, 30 to 50 years, 51 to 65 years, >65 years), sex (male, female), education level (8th grade or less, Advanced degree (eg, MA MS PhD MD), College degree (BA/BS), High school graduate/GED, Some college/Technical degree/AA/AS, Some high school) were compared using a Wilcoxon signed-rank test. All data analyses were performed using SPSS 23.0.¹⁰

Results

Demographic and Clinical Characteristics

The average age of participants recruited to take part in the study was 48.26 (SD = 14.25) years. A majority of participants were female (n=473; 73.9%). Participant demographic and clinical characteristics are presented in Table 2. Additionally, 35.4% of the sample reported they have participated in clinical trials (n = 221).

Table 2 Demographic and Clinical Characteristics of Participants

Device Use

A majority of participants, 67.0% (n = 429) indicated that they regularly use accessibility features on using smartphones and tablets. When prompted, 38.6% (n = 247) indicated they would like to see more than one item per screen but only if they are related and to see all text in a larger size with scrolling features using fingers (n = 246; 38.4%). In terms of font size, most individuals preferred Option B, in which handheld device text was displayed in 16 pt. font. Table 3 presents the participants’ responses to device use and functionality features they prefer to see on ePROs. There were no significant differences observed among sex, level of education, or age groups for the device features except for more participants in the 31 to 50 age group preferred 16 pt. font size compared to other age groups (p = 0.002).

Table 3 Participant’s Perceptions on ePRO Technological Features

Situational Questions

Responses to situational questions are presented in Table 4. Results show that most participants (n = 460, 71.9%) were able to accurately identify a seizure. More essential, a large majority of participants felt it would be beneficial to have a short training on seizure symptoms prior to completing a daily seizure diary (n = 552, 86.3%), and seizures should be reasonably reported within 10 mins (n = 426, 66.6%). Participants endorsed the presentation of all symptoms reported in the questionnaire, with repetitive movements and loss of consciousness as the most predominant symptoms they would look for (see Table 4).

Table 4 Participant’s Response to Situational Questions on Seizures Related Disorders

Discussion

Although there is evidence that electronic data capture on mobile devices (eg, ecological momentary assessment) is well accepted, participant perceptions in what they would like to see and find useful on ePROs have not been assessed. For many clinical trials, information directly from patients and their caregivers during scheduled site visits presents the most viable option of data collection (practical reasons, ensuring standardization of timing of assessments, etc.). The FDA has asserted its support and expectations for electronic capture of clinical trial source data, including PRO endpoints.¹¹ Additionally, ISPOR has launched three task forces that have issued ePRO-related good research practice (GCP) recommendations.^5,12 It is important, therefore, to assess ePROs for PCOR (and specifically the usability and features of electronic devices) used to collect data in such instances. ePROs have the potential to improve the quality and patient-centeredness of medical care in a variety of ways. They can be used at the individual patient level to improve interactions between patients and clinicians. They can also be used in research studies to identify the benefits and harms of interventions, especially in seizure-related disorders. Moreover, participants have a role to play in policy-making and population surveillance, including contributing to guideline development, informing coverage and reimbursement decisions, evaluating care quality, and identifying the impacts of policy options for seizure-related disorders.

Findings from the current study provide evidence that participants and caregivers can complete and are familiar with the use of electronic devices regardless of age, sex or education level. Additionally, participants and caregivers have important insights and discernments on both technological device use and seizure-related observations. These findings support those from previous studies showing no difference in ease of completion of ePROs according to participants’ familiarity of using computers or similar electronic devices.¹³ Furthermore, no differences in ease of completion based on functionality according to respondent age, sex, and education were observed. Observations from this study highlight several considerations for future use of ePROs. Findings from this study highlight the importance of when seizure symptoms can be accurately reported, perceptions that training on seizure diaries would be beneficial, and participants and/or caregivers’ preference to reporting symptoms in their own words rather than medical terms. All of patient-centered situational responses will help to guide development of seizure diaries, ePROs and technological device development to allow for more efficient and precise measurement.

Participants preferred to see individual items that were related to each other on the same screen. Gwaltney et al¹⁴ raised similar concerns, that having one item per screen removes the option for participants to review all their answers. Including participants in the decision-making process and providing information regarding their concerns is important for achieving compliance and usability. Kusec et al¹⁵ noted that participant involvement is essential in establishing methods for developing educational materials to improve patients’ understanding of procedures. In our study, having participants included in the decision-making process for further development of ePROs for seizure-related disorders provide valuable viewpoints. Finally, participants reported that education on the ePRO would be beneficial. Patient-reported outcomes (PRO) are subjective by design, as they are intended to measure the unique experience of each individual being assessed. However, this subjectivity also creates a measurement challenge, in that PROs tend to be vulnerable to pre-existing subject biases, as well as variability in their observation and/or interpretation of symptoms. One of the deterrents to reliable PRO data is inconsistency in the way subjects conceptualize the constructs and symptoms being measured. The interpretation of a specific clinical event may vary from subject to subject, and the same clinical event can be assigned different severity ratings within and across subjects. An additional factor in the quality of PRO data is lack of understanding of the measurement parameters and scoring anchors. For example, ePROs may vary in the specific symptoms they are capturing, time frame being evaluated (past week, daily), as well as in the scale dimension (intensity, frequency, impact, duration). It is important to explain these parameters to study participants to promote measurement accuracy and consistency. In this era of patient-centered care, a broad aim of patient education is to encourage individuals to actively participate in their own healthcare.¹⁶ Key features in completion and success of clinical trials are timely recruitment, compliance (to study procedures, rating scale completion, study drug and visits) and participant retention throughout the study.¹⁷ It follows that effective education/training of participants can positively affect these key features.

Limitations

There are a few limitations of this novel approach to ePRO development in seizure disorders. Firstly, only 5 participants had epilepsy or seizure-related disorders making it difficult to compare responses to participants with seizures and those without. Also, it may be difficult to ascertain if individuals with other medical or CNS disorders also experience seizures as that question was not asked as part of the questionnaire. Secondly, the study did not investigate the feasibility of utilizing more advanced mobile technologies (eg, participants’ own personal tablets and smartphones) to collect PRO data in participants (patient and their caregivers) on devices that are already familiar to them. Thirdly, findings from the study did not assess repetition of recall period on each screen, as incorrect recall has previously been demonstrated in both paper and electronically administered clinical outcomes assessments.¹⁸ Incorrect use of recall presents challenges to assessing changes on PRO parameters over time and should be considered when developing self-reported measurements.

Conclusion

The movement toward ePRO data collection (for existing or new measurements, or for seizure diaries) has been a significant advancement in self-reported measurement. ePROs result in more complete and accurate datasets, which could be the distinction between failed and successful clinical trials. Providing that the instruments are laid out clearly, instructions are easily understood, and the device is easy to use based on participant perceptions and needs, ePRO is a feasible method of data collection in patients and caregivers. This appears to be true regardless of demographic characteristics of participants.

For the future of data collection, research and policy initiatives are vital to facilitate the routine use of PROs in policy, clinical trials, and clinical practice, and to enable the association of ePRO data with clinical outcomes assessments and other clinical data to support PCOR.¹⁹ Therefore, it is important to develop ePROs that are agreeable and satisfy the needs of the participants and caregivers without compromising the scientific and clinical aspects of the symptoms or disease construct. Developing tools using participant needs, observations, characteristics and input is essential to putting the participant perspective in patient-centered outcomes research.