Diarrhea-predominant irritable bowel syndrome: creation of an electronic version of a patient-reported outcome instrument by conversion from a pen-and-paper version and evaluation of their equivalence
Received 4 November 2016
Accepted for publication 15 March 2017
Published 18 July 2017 Volume 2017:8 Pages 83—95
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 2
Editor who approved publication: Dr Liana Bruce
Leticia Delgado-Herrera,1 Benjamin Banderas,2 Oluwafunke Ojo,2 Ritesh Kothari,3 Bernhardt Zeiher1
1Astellas Pharma Global Development, Inc., Northbrook, IL, 2Adelphi Values LLC, Boston, MA, 3ACCESS Medical LLC, Chicago, IL, USA
Background: Subjects with diarrhea-predominant irritable bowel syndrome (IBS-D) experience abdominal cramping, bloating, pressure, and pain. Due to an absence of clinical biomarkers for IBS-D severity, evaluation of clinical therapy benefits depends on valid and reliable symptom assessments. A patient-reported outcome (PRO) instrument has been developed, comprising of two questionnaires – the IBS-D Daily Symptom Diary and IBS-D Symptom Event Log – suitable for clinical trials and real-world settings. This program aimed to support instrument conversion from pen-and-paper to electronic format.
Materials and methods: Digital technology (Android/iOS) and a traditional mode of administration study in the target population were used to migrate or convert the validated PRO IBS-D pen-and-paper measure to an electronic format. Equivalence interviews, conducted in three waves, each had three parts: 1) conceptual equivalence testing between formats, 2) electronic-version report-history cognitive debriefing, and 3) electronic version usability evaluation. After each interview wave, preliminary analyses were conducted and modifications made to the electronic version, before the next wave. Final revisions were based on a full analysis of equivalence interviews. The final analysis evaluated subjects’ ability to read, understand, and provide meaningful responses to the instruments across both formats. Responses were classified according to conceptual equivalence between formats and mobile-format usability assessed with a questionnaire and open-ended probes.
Results: Equivalence interviews (n=25) demonstrated conceptual equivalence between formats. Mobile-application cognitive debriefing showed some subjects experienced difficulty with font/screen visibility and understanding or reading some report-history charts and summary screens. To address difficulties, minor revisions/modifications were made and landscape orientation and zoom-in/zoom-out features incorporated.
Conclusion: This study indicates that the two administration modes are conceptually equivalent. Since both formats are conceptually equivalent, both are psychometrically reliable, as established in the pen-and-paper version. Subjects found both mobile applications (Android/iOS) offered many advantages over the paper version, such as real-time assessment of their experience.
Keywords: irritable bowel syndrome diarrhea-predominant, IBS-D, symptoms, mobile application, patient-reported outcomes, conceptual equivalence
Irritable bowel syndrome with diarrhea-predominant symptoms (IBS-D) has a major impact on patients, especially for those with moderate-to-severe IBS-D. It affects the lower gastrointestinal tract, causing such symptoms as abdominal cramping, bloating, pressure, and pain, although it does not cause permanent damage. Subjects’ health-related quality of life is substantially impaired, and leads to high health care-resource utilization. Because of an absence of established clinical biomarkers for assessing IBS-D severity,1 evaluation of the level of clinical benefit provided by a treatment is dependent on the availability of valid and reliable symptom assessments.
Patient-reported outcome (PRO) measures are useful for assessing the impact of a disease on patients, especially in situations where clinical determinations are as yet unavailable, as is the case with IBS-D. In order to improve PRO-measure effectiveness and availability, PRO instruments are being transitioned from traditional pen-and-paper formats onto electronic formats, administered via computers or other portable electronic devices. The use of electronic formats for collecting data has several benefits over pen-and-paper formats, eg, reducing missing or ambiguous data by not allowing the patient to move forward to the next question until the current one is answered and enabling the use of adaptive questioning, whereby the response from one question leads to the following question. In addition, transcription errors and the time taken to migrate or convert data from paper for analysis are eliminated. Electronic formats also allow data to be summarized in real time via graphs and tables, allowing quick access to patient data, which might yield information useful in guiding patient care. Lastly, since electronic devices are able to capture input times, patients’ compliance with completing electronic diaries and logs in a timely manner can be assessed.2,3
It is important to note what differences, if any, there are in subjects’ understanding and comprehension of an instrument in one format versus the other. Although the item content and response are identical between the pen-and-paper and electronic formats, they are presented and administered differently. It is also important to assess how easy it is for subjects to complete an instrument on an electronic device, particularly for those who may not be familiar with this platform.
The IBS-D PRO instrument, which comprises two questionnaires – the IBS-D Daily Symptom Diary and the IBS-D Symptom Event Log (Astellas Pharma Global Development, Inc., Northbrook, IL) was developed for use within clinical trials and real-world settings;4 its ability to be translated into ten languages has been confirmed.5 The primary goal of this study was to support the migration of the IBS-D PRO instrument from its pen-and-paper-based version to an electronic mobile version by testing the conceptual equivalence of the two formats and the usability of the mobile application for subjects with IBS-D.
Materials and methods
IBS-D PRO instrument
The IBS-D PRO instrument consists of two parts: the IBS-D Daily Symptom Diary and the IBS-D Symptom Event Log. It has undergone both qualitative assessments of content validity and quantitative assessments of its psychometric properties.5–7 The pen-and-paper version of the instrument is shown in Figures 1 and 2.
Figure 1 The six-item IBS-D Daily Symptom Diary.
Note: ©2012 Astellas Pharma Global Development, Inc. (APGD). Reprinted in Patient Related Outcome Measures with permission of APGD.
Figure 2 The four-item IBS-D Symptom Event Log.
Note: ©2012 Astellas Pharma Global Development, Inc. (APGD). Reprinted in Patient Related Outcome Measures with permission of APGD.
IBS-D electronic PRO instrument
The IBS-D electronic PRO (IBS-D e-PRO) instrument was developed for use on Android and iOS mobile platforms. It was derived from the original pen-and-paper version (IBS-D PRO), with careful consideration of ways in which patient interpretation of the instruments may differ between formats. As with the pen-and-paper version, the electronic version comprises both the IBS-D Daily Symptom Diary and the IBS-D Symptom Event Log (Figure 3). The electronic version of the IBS-D Daily Symptom Diary displays each of the six items on an individual screen to aid patient understanding and interpretation of the information (Figure 4). Likewise, the electronic version of the IBS-D Symptom Event Log displays each of the four items on an individual screen (Figure 5).
The electronic format allows inclusion of features not possible with the pen-and-paper format. The IBS-D e-PRO mobile application sends daily notifications via a pop-up screen to users to encourage entering of data in a timely manner. If data are not entered in the IBS-D Symptom Event Log within 24 hours, a reminder is generated by the application. Similarly, the IBS-D Daily Symptom Diary must be completed each day to allow entry of data into the IBS-D Symptom Event Log on subsequent days. Users are required to enter complete information for both the IBS-D Daily Symptom Diary and IBS-D Symptom Event Log. The report-history section allows aggregated data for periods of 1 day, 1 week, 1 month, and 3 months to be displayed in both graphic and summary format (Figures 6 and 7), with the option to share with a clinician via email as a PDF file attachment. For this study, sample data were programmed into the device, in order to evaluate the report-history section.
The IBS-D e-PRO instrument includes features ensuring data integrity and security when used in the clinical trial setting. Once subjects grant informed consent, each is assigned an identification number (for trial site access to data via a secured web portal), a randomization code (not available to subjects), and a four-digit access code (for subject access). Trial sites have the ability to access daily subject data and confirm integrity according to US Food and Drug Administration compliance, including guidelines of part 11 of title 21 of the Code of Federal Regulations (electronic records, electronic signature compliance), with an audit trail saved for data-change management. Study sponsors can access only aggregated data to determine compliance with the study protocol. Although not part of the application-development activities, the data-integrity and security features are also available when used in the real-world setting, as applicable Health Insurance Portability and Accountability Act law will be applied by physicians or clinicians when patients share data with them.
This study was conducted through three waves of equivalence interviews. If preliminary results from each wave of interviews warranted modifications to the mobile application, revisions were made and the revised application used with subjects in the subsequent waves of equivalence interviews (Figure 8).
Figure 8 Overview of methodology.
Prior to the study start, all required study documents were reviewed and approved by the Copernicus Group Independent Review Board (Durham, NC, US; protocol DV6442A). All study subjects provided written informed consent for study participation. All interviews were conducted face to face by trained personnel, who had undergone National Institutes of Health human participant-protection training, as well as data-protection and interview training.
Subjects and recruitment targets
Subjects were aged ≥18 years, with a clinician-confirmed diagnosis of IBS-D (mild, moderate, or severe) and symptoms for ≥6 months prior to diagnosis. To be eligible for the study, subjects had to meet all inclusion criteria and no exclusion criteria (Table S1). A priori recruitment targets were identified in order to achieve a diverse study sample and ensure the sample closely reflected the disease epidemiology (Table S2). The target enrolment was 25 subjects.
Interviews were conducted via 90-minute face-to-face interviews by trained personnel. Prior to study-subject interviews, mock interviews were conducted among interviewers to identify potential issues with the interview guide and to ensure good interviewing practices. All interviews with subjects were audio-recorded and transcribed by an independent transcription company. Transcripts were anonymized by removing potentially identifying information. Interviews consisted of three parts: 1) conceptual equivalence testing of the electronic versus pen-and-paper versions, 2) cognitive debriefing assessing subjects’ ability to understand and interpret the report-history section of the mobile application, and 3) evaluation of the usability of the mobile application.
Conceptual equivalence testing
Conceptual equivalence interviews aimed to evaluate the extent to which the migrated instrument (IBS-D Symptom Event Log and the IBS-D Daily Symptom Diary) was interpreted in similar ways by subjects, regardless of format. Subjects first completed both the mobile-application and pen-and-paper versions of the IBS-D Symptom Event Log and provided feedback, which included interpretation of the instructions, items, and response options in the questionnaires. Half of the subjects completed the pen-and-paper format first, while the other half completed the mobile-application format first. After completion of the first format, feedback was provided before starting the other format and subsequently providing its feedback. Subjects then completed the IBS-D Daily Symptom Diary in the same crossover manner. Before completing the electronic format, subjects were asked to download the mobile application onto their electronic device (with assistance from the interviewer if required). Assistance provided for the download was recorded.
Subjects were asked to use a “think-aloud” methodology, whereby they discussed the process used to arrive at their response to each questionnaire item, as well as how they interpreted each concept/item and response option in the questionnaires. The interviewers used targeted probes when feedback from the think-aloud method was incomplete or lacking. Subjects’ interpretations were compared across the two formats to assess whether the instructions, items, and response options were conceptually equivalent.
Cognitive debriefing for the mobile-application report-history section
The report-history section of the interview was a cognitive debriefing, which assessed subjects’ ability to understand and interpret results from the IBS-D Daily Symptom Diary covering periods of 1 week and 1 and 3 months, and from the IBS-D Symptom Event Log covering periods of 1 day and 1 and 3 months. Subjects were told that the report history contained sample data that did not reflect their responses to the questionnaires in the previous section of the interview (ie, the IBS-D Daily Symptom Diary and IBS-D Symptom Event Log). They were asked to interpret graphic representations of simulated data, and their understanding of the summary results for each item in the questionnaires was documented.
Usability of the mobile application
The usability section of the interview elicited subjects’ opinions on the mobile application’s ease of use, appearance, and downloading. Subjects could elaborate on their responses through open-ended questions posed by the interviewer, and were also given the opportunity to suggest any changes to improve usability.
Data handling, analysis, and presentation
Before complete coding and analysis of interview transcripts, interim results for each wave were created, based on the interviewers’ summaries of interviews, to identify equivalence, cognitive, or usability issues raised during that wave. Results from each wave were reported to the study sponsor and mobile-application developer, so that appropriate changes could be made before the next wave of interviews, if applicable.
Transcription and coding
Following the completion of the third wave of interviews, coding and analysis of the anonymized interview transcripts across all three waves were carried out. These anonymized transcripts served as the source data for analysis. Coding was used to organize and catalog subjects’ feedback during the think-aloud exercise and their responses to the open-ended interview questions. Transcripts were entered into Atlas.ti version 7.5 (Scientific Software Development GmbH, Berlin, Germany), a software package designed specifically to facilitate the organization and analysis of qualitative data.8
The primary goal of transcript coding for the conceptual equivalence section of the interviews was to organize and catalog subjects’ interpretations of the instructions, items, and response options that compose the instrument for each format. Conceptual equivalence data for each questionnaire in the IBS-D PRO instrument (ie, the IBS-D Daily Symptom Diary and the IBS-D Symptom Event Log) were analyzed separately. For the report-history section of the mobile application, transcripts were coded to organize and catalog subjects’ interpretations of the filters (ie, 1 day, 1 week, 1 month, and 3 months), charts, and summary information presented. The usability section of the transcripts was coded to organize and catalog subjects’ overall impressions of the mobile application, as well as their responses to specific questions related to download and installation, the appearance of text and icons, ease of response selection, and navigation through the mobile application.
For the conceptual equivalence data, subjects’ reports on interpretation of instructions, items, and response choices were compared between formats for each questionnaire. A determination was then made by the research team as to whether a subject interpreted each instruction, item, and response choice equivalently between the two formats. Determining conceptual equivalence was guided by the principles detailed in Table 1.
Table 1 Authors’ conceptual equivalence terminology
For the report-history section, subjects were deemed to have been able to interpret an item’s report history if they were able correctly to express the significance of an axis or point in a chart and the meaning of data contained in the item’s summary screen. Findings from the mobile application-usability section of the interviews, specific to interviewer questions regarding subjects’ ability to operate, maneuver in, and complete the mobile-application format of the instrument, were classified, organized, and analyzed.
Study sites and subjects’ demographic, medical, and health-related characteristics
In total, 25 subjects were recruited for interview from three clinical sites in the US. The first wave of interviews was conducted in Chicago, IL with seven subjects, the second wave in St Louis, MO with eight subjects, and the third wave in New Orleans, LA with ten subjects. Subjects’ demographic characteristics are provided in Table 2.
Table 2 Demographic characteristics of the study subjects
Subjects’ medical and health-related characteristics are provided in Table 3. As assessed by clinicians, subjects were evenly divided among levels of IBS-D severity (mild, eight [32%] subjects; moderate, nine [36%] subjects; severe, eight [32%] subjects). Most subjects considered themselves to be in either very good (n=9 [36%]) or good (n=9 [36%]) health, and more than half reported no coexisting health conditions (n=15 [60%]).
Table 3 Medical and health-related characteristics of the study subjects
Notes: Counts not mutually exclusive; one subject did not provide information on other health conditions.
Mobile-device operating systems
Of the 25 subjects interviewed, 16 (64%) used an Android mobile device and nine (36%) used an iOS device. Because of difficulties with downloading and installing the mobile application, eleven (44%) subjects who originally intended to use an Android device for the interview actually used a backup device (Android [n=3]; iOS [n=8]) provided by researchers conducting the interviews.
The interim results were summarized after each wave, based on interviewer notes, and appropriate changes to the mobile application were made prior to the next wave of interviews. A description of the results of each wave of interviews and modifications made after wave 1 and 2 interviews are provided in Tables S3–S5. Revisions were made to the mobile application to improve usability. No revisions were made to the instructions, items’ content, or response options of the IBS-D Daily Symptom Diary or IBS-D Symptom Event Log between waves of interviews or after the final wave of interviews.
Conceptual equivalence testing
IBS-D Daily Symptom Diary
The results of the equivalence interviews demonstrated the conceptual equivalence of the instructions, items, and response options of the IBS-D Daily Symptom Diary across both formats, with no subject providing interpretation data that were considered not conceptually equivalent. Nearly all subjects interpreted the instructions consistently across both formats (n=25 [100%] for the first part of the instructions [“The purpose of this diary . . .”] and n=24 [96%] for the second part of the instructions [“Please answer the following questions . . .”]); one subject (4%) provided insufficient data to determine conceptual equivalence for the second part. Most subjects (n=25 [100%] for items 1 and 3–6, n=24 [96%] for item 2) interpreted the items consistently across both formats, with one subject (4%) providing insufficient data to determine conceptual equivalence for item 2. Of the subjects who provided an interpretation, most (≥20 [≥80%]) interpreted the response options to all items consistently across both formats (Table S6).
IBS-D Symptom Event Log
The results of the equivalence interviews demonstrated the conceptual equivalence of the instructions, items, and response options of the IBS-D Symptom Event Log across both formats, with no subject providing interpretation data that were considered not conceptually equivalent. All subjects interpreted the instructions and all four questionnaire items consistently across both formats, and most subjects who provided an interpretation (≥22 [≥88%]) interpreted the response options consistently across formats (Table S7).
Cognitive debriefing of the mobile-application report-history section
IBS-D Daily Symptom Diary report history
All subjects who provided an interpretation of the IBS-D Daily Symptom Diary report-history instructions interpreted them as intended. No revisions to the mobile-application instructions were suggested. Subjects’ interpretations of the mobile-application report history for the IBS-D Daily Symptom Diary are shown in Table S8. Revisions to the “how often passed gas” chart and summary screen, “number of days without accidents” chart, and “average severity of IBS-D pain, cramping, and pressure” chart and summary screen were recommended (Table 4).
Table 4 Final revisions to the mobile application following completion of all three interview waves
Abbreviation: IBS-D, irritable bowel syndrome with diarrhea.
IBS-D Symptom Event Log report history
Subjects’ interpretations of the mobile-application report history for the IBS-D Symptom Event Log are shown in Table 5. No revision to immediacy-of-need or bowel-movement-appearance report-history charts was recommended, but revisions were recommended for their respective associated summary screens (Table 4). As subjects completed each questionnaire (ie, the IBS-D Daily Symptom Diary and the IBS-D Symptom Event Log) in the mobile application, a pop-up screen appeared with the following message: “Thank you for your entry. Click ‘Submit’ if you are done. Click ‘Close’ and then Click ‘Previous’ if you would like to review your answers”. Subjects were asked their interpretation of this pop-up notification following their completion of the IBS-D Symptom Event Log. All 24 subjects who provided a response interpreted the pop-up notification as intended; due to data limitations, data for one subject (4%) were not collected.
Table 5 Interpretation of the mobile application IBS-D Symptom Event Log report history
Abbreviation: IBS-D, irritable bowel syndrome with diarrhea.
Usability of the mobile application
All 25 subjects (100%) had a positive overall opinion of the mobile application, with some describing it as “convenient”, “user-friendly”, “clear”, “helpful”, and “comprehensive”. Without interviewer prompting, 18 (72%) subjects indicated that the application was easy to use, and three (12%) indicated that it would be easy or beneficial for their physician to review the data. Changes to the application were suggested by seven (28%) subjects, the majority of which were related to its appearance to make it easier to read and understand. Usability-interview responses are summarized in Table 6 and revisions in Table 4.
Table 6 Mobile-application usability interview responses
Final revisions to the mobile application
Equivalence interviews (n=25) demonstrated conceptual equivalence between formats. To address some difficulties in the report-history section, which is not part of the pen-and-paper version, some revisions were made only to this section of the mobile application in order to improve screen visibility. Revisions to the report-history section included minor modifications to the font size, landscape orientation, and zoom-in and zoom-out features, which improved the usability and visibility of this section. Final revisions are shown in Table 4.
The primary goal of the mode-of-administration equivalence interviews was to test the conceptual equivalence between the pen-and-paper and mobile-application formats of the IBS-D PRO instrument. The interviews demonstrated that pen-and-paper and mobile electronic formats of the IBS-D PRO instrument were conceptually equivalent. Overall, subjects interpreted the instructions, items, and response options consistently across both formats.
An additional goal of the equivalence interviews was to conduct a cognitive debriefing of the report-history section of the mobile application, a feature not feasible in the pen-and-paper version. In its real-world setting, the report-history section will allow subjects to view a summary of their responses to each item in both questionnaires over a selected period of time (past week, month, or 3 months for the IBS-D Daily Symptom Diary; past day, month, or 3 months for the IBS-D Symptom Event Log). The report-history section was populated with sample data, and subjects were asked to provide their interpretation and understanding of the chart and associated summary screen for each item of both questionnaires of the IBS-D PRO instrument. For the IBS-D Daily Symptom Diary report history, while subjects were able to interpret and understand the charts and summary screens for the first four questionnaire items as intended, some subjects experienced difficulty in interpreting and understanding the other two items; where applicable, appropriate revisions were made to the charts and summary screens. Several subjects also experienced difficulty understanding and interpreting the average of the abdominal pain, cramps, and swelling score chart and its associated summary screen; minor revisions were made to both. For the IBS-D Symptom Event Log report history, subjects were able to understand and interpret the report-history charts for every item as intended. However, some did not interpret the summary screens for the “immediacy of need” report history and the “bowel movement appearance” report history as intended; changes to both of those summary screens were made.
The final goal of the equivalence interviews was to obtain subjects’ opinions on the mobile application and its usability. All subjects expressed a positive opinion of the mobile application, with many spontaneously noting that it was easy to use and some reporting that they preferred the mobile application to the pen-and-paper format. Some subjects also felt that the mobile-application report history would be easy or beneficial for their physician to review their data. However, subjects expressed difficulties with two aspects of the mobile application. First, about a third of subjects noted that the small font made questions and charts on the mobile application difficult to read. Landscape orientation and zoom-in and zoom-out features were incorporated to make the text and charts clearer. Secondly, about half of the subjects experienced difficulty in downloading and installing the mobile application on their devices. This was true for users of both Android and iOS devices. However, download issues are expected to be obviated when the mobile application becomes publicly available in the application stores of both mobile platforms.
In conclusion, this study demonstrates the equivalence of the pen-and-paper and mobile-application formats of the IBS-D PRO instrument. Usability testing also confirmed that the mobile application was, by and large, used easily by subjects, with no difference observed in usability by age. The subjects found that both mobile applications (Android and iOS) would offer many advantages over the original paper version, such as real-time assessment of patients’ experience by physicians. Given the equivalence and past established measurement properties (ie, reliability and validity) of the IBS-D PRO instrument, use of the electronic administration mode is recommended, given its ability to capture higher-quality data versus the original pen-and-paper version.
The PRO instrument for IBS-D was developed by Astellas Pharma Global Development, Inc., and the electronic version was developed by ACCESS Medical LLC. The authors would like to thank Brittany Klooster, MPH and Roger Lamoureux, MDiv at Adelphi Values LLC for conducting interviews and data analysis. The authors would also like to thank Jane M Gilbert, BSc, CMPP and Gill Sperrin, CBiol, MRSB, CMPP at Envision Scientific Solutions for providing medical writing support, funded by Astellas Pharma Global Development, Inc. This study and the preparation of this paper were funded in full by Astellas Pharma Global Development Inc.
LDH, BB, OO, RK, and BZ contributed to the design of the study, RK performed the research and designed and developed the mobile application, BB and OO performed the research and participated in data acquisition, and all authors were involved in the analysis/interpretation of the data and drafting or critically reviewing/revising the manuscript. All authors approved the final version of the manuscript, and agree to be accountable for all aspects of the work.
LDH and BZ are employees of Astellas Pharma Global Development, Inc. BB and OO are employed by Adelphi Values, and were contracted by Astellas Pharma Global Development, Inc. to work on IBS-D PRO instrument development. RK is employed by ACCESS Medical LLC, and was contracted by Astellas Pharma Global Development Inc. to migrate the IBS-D PRO instrument from a paper-based version to an electronic version for use on mobile electronic devices. The authors report no other conflicts of interest in this work.
This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.Download Article [PDF]