Opportunities and Challenges of Web-Based and Remotely Administered Surveys for Patient Preference Studies in a Vulnerable Population

Overview

In a collective effort to move towards a more patient-centric approach to medicinal product development,^1–3 drug companies, regulators and assessment bodies are increasingly seeking to incorporate patients’ preferences regarding therapeutic options and attributes as a formal element in their decision-making regarding the benefit-risk profile of a product. Among the key drivers of this change is the notion that patients are not only the ones who directly benefit from a therapy but also the ones who bear its burden in that they are exposed to the treatment’s side effects.⁴ Patient preference studies capture patients’ views on which treatment and disease-related characteristics are important to them, the trade-offs they are willing to make between positive and negative aspects of different therapies and treatment attributes, and the impact of individual patient characteristics on the formation of such preferences.^5,6 The present commentary discusses opportunities for conducting patient preferences studies using web-based modes of data collection in the context of the Patient Preference in Benefit-Risk Assessments during the Drug Life Cycle (PREFER) project. As part of the Innovative Medicines Initiative (Horizon 2020), this 5-year research project aims to develop evidence-based recommendations to guide industry, regulatory authorities, health technology assessment (HTA) bodies, and other researchers on when and how patient preference studies could be used to inform decision-making throughout the medical product lifecycle (for an overview, see https://www.imi-prefer.eu/ or https://www.imi.europa.eu/projects-results/project-factsheets/prefer). The project features three patient preference core case studies within three disease areas: non-small-cells lung cancer (NSCLC), rheumatology, and neuromuscular disorders.

The application of web-based surveys administered via use of a remote modality is becoming increasingly popular due to the fact that it offers numerous advantages over traditional paper-based or computer-assisted surveys.⁷ During the COVID-19 pandemic, clinical studies were forced to rely more on remotely administered online surveys to gather data than initially planned. However, it is unclear whether complex preference elicitation tasks administered online in highly vulnerable patient populations are also feasible.

In order to discuss the feasibility of web-based preference surveys in vulnerable populations, we refer to the non-small cell lung cancer (NSCLC) quantitative study being conducted as a part of PREFER. This case study involved a large cohort of patients with NSCLC at stages from I to IV recruited from three cancer treatment centers across Italy and Belgium.⁸ The clinical aim of this case study was to assess how patients trade-off benefits and risks related to treatment alternatives, by using both discrete choice experiment (DCE) and swing weighting (SW) methods. DCE asks respondents to make choices among sets of profiles in a series of choice questions, with different levels of benefits and risks associated with the choices. A typical DCE question is “Which of the (two) treatments do you prefer?”. Conversely, SW asks patients to rank and rate different treatment attributes, and the preferences that inform this decision are assumed to be directly captured with the elicitation task.

Lung cancer patients represent a vulnerable population in that they tend to be older (65 years of age or older at the time of initial diagnosis) and report high levels of physical and psychological morbidities associated with their disease.⁹ In this paper, we describe opportunities and challenges associated with conducting web-based preference studies derived specifically from the PREFER Lung Cancer case study.

Web-Based Data Collection in Patient Preference Studies: General Methodological Considerations

While preference elicitation tasks have traditionally been administered in paper- or computer-based formats requiring the researcher’s presence (for review see¹⁰), the last 10 years have seen a rapid increase in the use of surveys and preference studies that are self-administered by participants online.⁷

Online data collection is a method that allows researchers to collect data from large and diverse populations in a quick and cost-effective manner.¹¹ Several studies have investigated the validity of web-based surveys by comparing the results of online studies with studies conducted via traditional modalities (eg, mail, in-person interviews). Results indicated that the validity and reliability of data obtained online are comparable to those obtained by classical methods.^12–14 However, there are several limitations that should be kept in mind when choosing a web-based survey approach for assessing patient preferences. One issue is the potential lack of inclusiveness and representativeness of this research method, whereby populations who do not have access to the internet and digital computing resources (eg, elderly or people with a low socio-economic status) may be underrepresented in the results.¹⁵ Some patient populations may live in rural contexts (with poor internet connectivity), maybe less educated, with lower literacy more in general. In those instances, an online survey would be an obstacle to their participation, whereas the presence of an experimenter helping them in understanding tasks and choosing responses to the survey plays a very important role. Another limitation is digital literacy and the age of the participants, whereby low digital skills and older age may limit participation in a self-administered web-based survey.^16,17 Web surveys must therefore be designed in such a way that they are simple to complete by patients with lower digital literacy, with instructions that are clearly stated so as to obviate the need for clarifications by the experimenter. An additional concern regards ethical matters and, particularly, challenges regarding participants’ consent, privacy limitations, anonymity and confidentiality posed by websites.

Given the numerous advantages of online surveys and the surge in the use of technology prompted by the recent COVID-19 pandemic, remote/online interviews and web-based surveys (“e-surveys”) are likely to become increasingly popular modes of administration. The present article contributes to the field by exploring this format for organizing and conducting patient preference studies in a frail and elderly population, namely lung cancer patients, involving virtual interactive content that can be administered online. This commentary focuses on two quantitative preference elicitation tasks used in the PREFER project (ie, DCE and SW), and highlights the strengths and limitations of web-based surveys administered remotely in a diseased population.

A Web-Based Implementation of Preference Surveys in an Elderly and Diseased Population

Methods Overview

Due to the SARS-CoV-2 pandemic, the original protocol of the PREFER lung cancer quantitative case study⁸ had to be modified. Originally, the protocol specified that eligible NSCLC participants could be contacted either by phone or approached in the waiting room either before or after a medical visit, at which point the researcher would introduce the study. In the original protocol it was anticipated that surveys were to be run on tablets during a clinic visit, and should last about 1.5 hours. In response to COVID-19-related restrictions regarding in-person contact, the protocol was adapted to an online mode of recruitment and to a web-based self-completed modality of survey administration. Recruitment had to be carried out primarily via telephone so as to limit physical contact between patients and researchers. Once patients had indicated interest in participating, they would receive an invitation letter via e-mail, as well as a participant information sheet, consent form, and contact information of the principal investigators. After returning the forms to the researcher, participants would then receive a personalized link to fill in the survey online. Upon entering the survey, patients were again probed to give their consent by selecting “Yes, I want to participate” in the corresponding informed consent online, and then asked to complete all the survey sections (without the presence of the researcher as planned in the original protocol). The project received ethical approval by the Ethische Commissie Onderzoek UZ/KU Leuven (reference S64022) in Belgium, and by the “Ethical Committee of the European Institute of Oncology IRCCS (reference R1142/20-IEO 1206) in Italy.

The platform chosen to host the survey was Sawtooth software,¹⁸ which handles both traditional survey questions and choice analysis. Within the survey, training material was provided in the form of interactive tutorials developed specifically for the case study that contained graphics, pictograms, icon arrays, voice-over, and click-on functions (for a recent description see¹⁹) (Figures 1A and B). This e-learning tool (referred to as an educational tool hereafter) introduced participants to general health-related terminology, as well as more specific disease-related content (eg, existing therapies for lung cancer), explained the basic aims of the choice tasks, and instructed participants as to how to perform the task and state their preferences. The tool introduced contents in a direct fashion using plain language and provided practical examples to complement the verbal instructions. In this way, the tool was intended to assist patients by increasing their functional health literacy (ie, the individual’s capacity to obtain, process, and understand basic health information and make appropriate health decisions critical to perform the preference elicitation tasks in our case), while explaining the task in simplified terms.

Figure 1 Illustration of PREFER survey. (A) Educational tool introducing discrete choice experiment (DCE); (B) educational tool introducing swing weighting (SW).

A first version of the survey including the educational tool was tested in a pilot study with a small group of patients (N = 5) who provided feedback regarding how understandable the material was and whether the survey length and burden to the patient was acceptable. The tool content and format were then revised according to this feedback. During the pilot testing, the researcher shared the screen with the participant and used the think aloud technique²⁰ to elicit thoughts or insights.

Opportunities and Challenges of Web-Based Survey for Preference Studies

Opportunities

The implementation of a web-based and remotely administered survey for the elicitation of patient preferences in PREFER in vulnerable populations had several advantages, which should be considered when preparing to conduct future studies of this type. First, as the result of using an online survey, we were able to enroll a larger portion of eligible patients, including those who reside in distant geographical areas and those who are physically and psychologically too frail to travel for research purposes, which constitute a large portion of our lung cancer (LC) patient population. Out of all LC patients enrolled in our study, roughly 39% lived outside the regions where the clinical centers were located – making it unlikely that they would have accepted to take part in a face-to-face study. Additionally, web-based questionnaires reduced the financial costs and time associated with using paper-based, computer-assisted approaches to survey administration,^21–23 thereby accelerating data collection. Indeed, we were able to recruit a sample of 307 patients over a 24-week period, a sample size that would have been very difficult to achieve if using a face-to-face design. Moreover, this modality reduced the burden on the patients – who did not need to spend additional time in the hospital beyond the clinical and therapeutic commitments, as well as on the researcher – who did not have to work around the patient schedule and be present during the survey completion.

On the design side of the research, web-based surveys allowed us to create complex yet easy-to-navigate branching and to embed multimodal material within a common e-platform. For instance, the PREFER survey included several educational videos aimed at facilitating patients’ comprehension of treatment attributes and their levels (Figure 1A and B), which were combined with DCE (Figure 2A), SW (Figure 2B and 2C) and Likert scale questions (Figure 3). Thus, the online format boosted the interactivity of the platform, while providing a user-friendly interface. With higher interactivity leading to a greater user engagement,^24,25 it is possible that highly interactive online surveys may be associated with greater levels of user satisfaction. In order to ensure that the survey structure and content could be comprehended without further clarifications by the experimenter, we used insights gained from the pilot study to identify and remove potential elements of confusion or biases introduced by the task instructions.

Figure 2 Illustration of PREFER survey. (A) DCE task probing participants to choose between two alternative treatment options given a set of attributes; (B) SW task requiring participants to sort attributes by importance, then (C) assign a weight (0–100) to each.

Figure 3 Illustration of PREFER survey. Likert scale questions were used to gather information regarding participants’ difficulty in understanding/answering questions.

With regard to the respondents’ experience, online formats allowed patients to work from a chosen “COVID-19 safe” environment where they might feel more comfortable compared to face-to-face setting. Additionally, this online format allows for an additional layer of flexibility in that users were able to exit the survey and re-enter the survey at a later time. This flexibility has two primary benefits. First, it reduces the burden to vulnerable populations who may become fatigued during surveys and need to stop. Since fatigue is a defining feature of many lung cancer treatments, this flexibility is necessary in order to reduce the chance of missing data or biased outcomes coming only from those with less severe illnesses who do not feel fatigued. Second, it allows respondents time to think through their responses when participating in a DCE, which enhances the validity of the elicited preferences.²⁶ Finally, we included “hover” functions throughout the survey to help patients understand complicated words and phrases so that they did not have to refer back to earlier sections in the survey for definitions of particular words and/or concepts.

Challenges

Despite the numerous advantages, we also experienced numerous challenges. A challenge associated with web-based surveys (and all research) is including older and vulnerable populations who may simply not have access to digital resources, because of lack of expertise in managing digital tools or lack of technologies. Such condition not only makes it difficult to recruit them (as many recruitment protocols use online channels) but also makes it difficult to reach them and have them complete the survey. The risk of collecting incomplete data, falsified answers, or low-quality answers should be also considered, even if these aspects are particularly frequent when studies are conducted in general population samples instead of patient populations. Strategies to engage patients and improve data collection may need to be employed in order to ensure that the preferences identified in these types of studies truly reflect the preferences of the patient population.^4,27 In our study, for instance, we tried to maximize the engagement of participants by highlighting the relevance of this study and the importance of including patients’ voice in the development of future therapies. Methods to increase the inclusiveness and representativeness of these studies should be the subject of further research.

Although the availability of digital technology has allowed researchers to surmount some of the difficulties introduced by social distancing, the greatest challenge in web-based research remains the poor digital literacy of elderly people, individuals who comprise a large portion of all NSCLC patients.²⁸ As technology becomes increasingly important in our society and more people embrace digital solutions,^29,30 it is possible that digital literacy will become less polarized to digital-native generations and more equally distributed throughout the population. However, as highlighted more than 20 years ago by Carbonaro and Bainbridge³¹ web-based surveys should only require a minimum of computer skills to ensure accessibility and ease of use. Thus, although future online research may be less affected by the participants’ age, researchers should continue to strive to design surveys with user-friendly interfaces. Initial feedback from a sub-sample of the target participant population is essential to ensure a close involvement of NSCLC patients in the design phase. To this end, we performed a pilot study on a small sub-sample of NSCLC patients, which allowed us to improve accessibility and user interface of the PREFER survey. For instance, we embedded the educational tool directly within the survey, rather than opting for the original pop-up solution (ie, where content is displayed in a separate window). The amount of information provided was reduced to a minimum to avoid distracting participants from the task at hand. We also improved the visual aspect by replacing text with graphic material whenever this was possible, and by using flashing indicators and instructions. Finally, we reduced the number of times participants had to interact with the survey.

Another limitation of web-based modes of testing is the lack of immediate access to a live individual for assistance in interpreting survey. This lack of access can reduce respondent engagement as compared to participating in a face-to-face interview.³² To minimize this risk, we included interactive educational videos, which provided relevant instructions and other information in plain language. To ensure the comprehensibility of the text the contents were revised by an expert linguist and professional scientific communicator. We also included contact details (telephone, email address) of the involved researchers for patients to call while completing the survey. Despite the apparent benefit for patients’ comprehension, future research is needed to assess the efficacy of these tools in compensating for the absence of the experimenter.

Moreover, it is possible that respondents feel less motivated to answer questions in a thoughtful manner if they are not in an official research context. They might struggle to understand the overarching aim and consequently they might fail to perceive the importance of their contribution.³³ Thus, web-based questionnaires may be more likely to be left incomplete and a higher number of patients might even decide to withdraw from the study. Indeed, in our LC study, a rate of 23.4% of the patients that initially gave their consent decided to withdraw from the study. To avoid this, as part of our protocol, we re-contacted patients who were flagged in the system as “incomplete” to resolve any technical issues and to encourage them to complete the survey. This allowed us to collect roughly 19.5% of the surveys that had been left incomplete.

A general limitation of online research is the lack of control over who is completing the survey. To minimize the risk that people other than NSCLC participated in the survey, we carefully planned the recruitment strategy such that participants were recruited exclusively from the three cancer centers across Italy and Belgium, closely involving and relying on input from clinical partners (oncologists and nurses) to select participants. A less controlled strategy (eg, recruitment through online patient panels or sending out survey links to cancer patient associations and groups) might have introduced a number of responses from participants who do not meet the inclusion criteria. It is also important to take into account the influence that family members or caregivers might have on the patients’ responses. We can presume that patients who are assisted during the completion of the questionnaire (eg, due to poor digital literacy or because they did not own a device themselves) may be influenced in their interpretation of the task by the helper’s own interpretation of the survey questions, raising the concerns that patients’ preferences expressed in the survey might not entirely reflect their own perspective. In face-to-face testing this is less likely to happen, since researchers are trained to be neutral when administering questionnaires. Future web-based research should aim to minimize this risk by first and foremost simplifying the survey as much as possible so that patients can complete the survey without help, and secondly by stressing the importance that those who assist them do not provide help beyond resolving technical issues. Additionally, feedback questions and evaluations could be included to gain insights into the influence of family members in completing the survey.

Finally, if data protection is always pivotal in research involving human participants, web-based surveys call for a special attention to the topic. Challenges regarding privacy limitations and confidentiality must be carefully addressed by researchers intending to carry out online research. For example, web surveys must have a built-in security system to ensure credibility and anonymity.³⁴ Although privacy protection and information security is pivotal for any online data collection, this certainly is the case for vulnerable patient groups, as was the case in the PREFER lung cancer study, who should be reassured that their identity and their medical information is protected and safely handled.

Conclusions

The present article discussed opportunities and challenges of web-based patient preference studies drawn from our recent experience with the PREFER quantitative project. This online and remote mode of administration had several important advantages over a face-to-face method of data collection, including the possibility of reaching a wider geographical area in a shorter timeframe, the limited burden to the patient, the greater flexibility offered to participants to complete the questionnaire from the environment of choice and in their own time, and the opportunity to include multimodal educational tools to improve communication with and comprehension by the patient participants. Several limitations were also identified and should be taken into account when designing future preference studies. Among these is the issue of a potential lack of inclusiveness, whereby only those that have (physical) access to web-based surveys can participate - this can in turn result in a limited representativeness of the research. The digital literacy level (and the literacy more in general) of potential participants needs to be carefully considered by simplifying as much as possible the survey format and questions, as low digital literacy may decrease the likelihood that participants agree to participate or that they are able to complete the survey without the help of others. Moreover, the inability to provide immediate feedback and clarify potential elements of confusion should be tackled in future preference studies, for example, by offering assistance via phone or video-call, as the additional difficulty for respondents may again lead to a lower participation of elderly patients or those who are not well versed in technology. Finally, it is possible that completing the survey remotely rather than in a physical research context may be associated with lower levels of engagement with the study. Thus, there are trade-offs to be made when choosing for either a remote or face-to-face study design. Future patient preference studies in vulnerable populations should directly compare the two modalities alone vs mixed-mode designs of data collection³⁵ in medical settings, evaluating response rates, the impact of the setting on patients’ engagement and potential for time and cost savings, to develop strategies to increase patients’ motivation to participate. Feedback with ad hoc questions on patients’ understanding of the study materials, their experience during the web-based surveys and the time to take the survey should be collected and analyzed in relation to their digital literacy, age and other factors that might influence the perceived difficulty of the survey. Furthermore, in order to improve the generalizability of the results obtained through a web-based mode of administration, future studies should examine other vulnerable populations beyond lung cancer patients.

In view of the current COVID-19 pandemic, the expansion of web-based methods and increasing recognition of their advantages, future studies should strive to address the limitations and opportunities identified in this paper to improve remote online methods, and adapt them to meet the needs of study participants. Experience from the PREFER lung cancer preference study, however, suggests that web-based quantitative surveys may provide a valuable method to explore patient preferences beyond the COVID-19 pandemic.