Evolution of the RebiSmart^® Electromechanical Autoinjector to Improve Usability in Support of Adherence to Subcutaneous Interferon Beta-1a Therapy for People Living with Multiple Sclerosis

Plain Language Summary

The RebiSmart electromechanical autoinjector is easy to use and helps people living with multiple sclerosis (MS) to take their treatment as prescribed. In this paper, we report on a next-generation device, RebiSmart 3.0, which was designed to meet their changing needs.

Initial studies looked at a prototype RebiSmart electromechanical autoinjector: its advantages, potential faults, and opportunities to improve it. These findings were incorporated into RebiSmart 3.0 for a follow-on study on how safe and easy it is to use. In the study, 45 participants (including adults and adolescents living with MS) completed a series of critical tasks necessary for the safe operation of the device. Participants then gave their own opinion of the device and completed a survey with questions on how easy and practical they thought the device was to use.

The test results showed that participants made mistakes that could be prevented by training, such as not inspecting or disposing of the cartridge properly and not washing their hands or disinfecting the injection site. Overall, 43 out of 45 participants considered the device as being safe to use. When asked about how easy and practical the device was to use, most participants gave the device the highest score possible. These results were used to update training materials and instructions for use.

In summary, the next-generation device, RebiSmart 3.0, is easy to use and no safety concerns were identified.

Introduction

The extent to which a patient follows the prescribed treatment (adherence) and the period of consistent use of the prescribed medical regimen (persistence) can have important health consequences for a wide variety of chronic illnesses. For multiple sclerosis (MS), which is a chronic and degenerative disorder of the central nervous system, better adherence to treatment with interferon-beta preparations by injection is associated with a lower risk of relapse.¹

To assist people living with MS with their adherence to such treatment, electromechanical autoinjectors have been developed to make the injection process more comfortable, easy, and reliable regarding dosage and timing of administration.² Subcutaneous interferon beta-1a (sc IFN β-1a), administered three times weekly, is a well-established and effective disease-modifying therapy for people living with MS and has an estimated cumulative exposure of 1,908,836 patient-years (to 30 September 2022). The RebiSmart electromechanical autoinjector – which includes the option to personalize injection attributes (injection depth, speed, and duration) and track injection history – was designed to be easy to use (as confirmed in several studies^3–5) and, in turn, supports their adherence to such therapy.^2,3,6–8 Indeed, the RebiSmart device may improve clinical outcomes through improved adherence to treatment with sc IFN β-1a.^9–12 However, it is important that the device continues to evolve over time to meet the changing needs of people living with MS. For example, earlier generations of the device have been updated on the basis of user feedback, which identified various design and usability opportunities for improvement (eg, refinement to on-screen instructions and those relating to cartridge expiry and use, as well as needle attachment and detachment).

We now report on the usability of an updated next-generation device, RebiSmart 3.0 (Figure 1), designed to assist people living with MS to adhere to their sc IFN β-1a therapy. A series of formative usability studies with a prototype device were initially conducted to identify the device’s strengths and weaknesses, and potential use errors. The findings were implemented into the design and use-related risk analysis of the updated device, RebiSmart 3.0, which was subsequently evaluated in a summative usability study that also incorporated a survey of user needs. The overall aim, therefore, was to validate the updated device as safe to use by intended users in the intended use environment.

Figure 1 The RebiSmart® 3.0 device is an electromechanical autoinjector to administer subcutaneous interferon beta-1a (Rebif®) for the treatment of patients with multiple sclerosis.

Methods

Formative Usability Studies

In early formative usability studies, 46 participants considered two electromechanical autoinjectors including the prototype RebiSmart device while an additional eight participants evaluated the instructions for use (IFU). These studies culminated in a final formative usability study in which a further nine participants, including adults living with MS (n = 3), adolescent proxy individuals (n = 3), and MS nurses (n = 3), evaluated the updated device and IFU. This study aimed to identify the device’s strengths and potential use difficulties, as well as determine recommendations for improvement. The findings and recommendations were subsequently incorporated into a use-related risk analysis for the next-generation device (RebiSmart 3.0), which was then evaluated through a summative usability study and subjective assessment as described below.

Summative Usability Study and Participants’ Subjective Assessment

In the summative usability study, adults living with MS (n = 12), adolescents living with MS (n = 3) and adolescent proxy individuals (n = 8), lay caregivers for someone with MS and proxy lay caregivers (n = 12), and healthcare professionals (n = 10) evaluated the RebiSmart 3.0 device. Evaluation activities were selected to validate the mitigations implemented to reduce the risks associated with not completing critical tasks essential for successful administration of the medication. Critical tasks were defined as tasks that, if performed incorrectly or not performed at all, would (or could) cause serious harm to the patient or user. Each critical task was evaluated through a sequence of simulated use scenarios and knowledge tasks. During each evaluation activity, recorded observations included all use errors, instances of moderator assistance, close calls, and difficulties during device operation. The evaluation activity performance data were consolidated and analyzed, focusing on the potential for serious harm arising from the critical tasks (where harm is defined to include compromised medical care).

After the participants had completed all use scenarios and knowledge tasks, the moderator asked each participant to provide their subjective impressions and feedback concerning the device and the instructional materials. In particular, the moderator asked each participant a series of open-ended questions as to whether they considered the product (ie, the autoinjector and instructional materials) safe to use as-is, or whether modifications were required to ensure its safe use.

User Needs Survey

Following their summative usability study session, the participants completed an evaluation of user needs and feedback statements through a series of questions, scored on a 5-point rating scale (1 being the lowest score; 5 being the highest score), to assess device usability and design. RebiSmart-experienced participants – existing users of the RebiSmart 2.0 electromechanical autoinjector – compared their current device against RebiSmart 3.0, while RebiSmart-naïve participants – having never used the RebiSmart 2.0 electromechanical autoinjector – only considered the new device.

Results

Formative Usability Studies

In the final formative usability study, nine participants evaluated the prototype device. Figure 2 summarizes the device’s strengths, opportunities for improvement, and the improvements that were to be implemented. Overall, the participants regarded the IFU as helpful, with opportunities for improvement including providing help with navigating the contents.

Figure 2 Strengths and opportunities for improvement of the prototype device.

All of the changes to the individual device functions that were considered to improve the prototype device and IFU are shown in Supplementary Table 1. These recommendations were used to inform the use-related risk analysis and improvements to the device, RebiSmart 3.0, which was then assessed in the summative usability study described below.

Summative Usability Study

In the summative usability study, 45 participants evaluated the updated RebiSmart 3.0 device.

Use Scenarios

The performance summary of the use scenarios, designed to ensure that all critical tasks were mitigated effectively, are summarized in Table 1. Specifically, the incidences of key use errors (incidences ≥10 are illustrated) associated with the critical tasks are detailed.

Table 1 Key Use Errors During Use Scenarios (Summative Usability Study)

Knowledge Tasks

The performance summary of the knowledge tasks is summarized in Table 2. Knowledge task 1 consisted of a knowledge check and a reading comprehension of the IFU, whereas knowledge task 2 concerned a reading comprehension of the on-screen instructions. To simulate the understanding of on-screen instructions for the purpose of the test session, participants were required to evaluate a paper-based test (a print of the screen text and image on paper). A summary of findings for all performance tasks evaluated is shown in Supplementary Table 2.

Table 2 Use Errors with Knowledge Tasks Related to Safe Use of the Device (Summative Usability Study)

Participants’ Subjective Assessment

For the participants’ subjective assessment of the updated device and the instructional resources, 43 out of 45 participants considered the product safe to use as-is. Modifications suggested by two participants for the device to be considered safe to use were to include a wrist strap to reduce the risk of the device being dropped (considering that the manual dexterity of a participant may change over time with reduced ability to grip the device); and to increase the clarity of the warning screens, such that the required action is clear.

User Needs Survey

The participants of the summative usability study completed a survey of user needs. The results of the average rating of participants’ scores across the 10 questions, and percentage of participants providing the highest ratings, are summarized in Table 3 and Figure 3, respectively.

Table 3 Summary of User Needs Findings (Summative Usability Study)

Figure 3 Percentage of participants providing the highest rating in the individual user needs survey (summative usability study).

Concerning the feedback statements, which were designed to estimate ease of use and comfort with the updated device, average scores were in the range of 4.5 to 4.9 out of 5 and up to 88% of participants provided the highest rating (Table 4). Responses to the feedback statements also showed a level of consistency between the scores awarded by RebiSmart-experienced and RebiSmart-naïve participants except for robustness, with a disparity of 3.4 versus 4.5, respectively (Table 5). Percentage of participants providing the highest ratings, according to prior experience of the RebiSmart electromechanical autoinjector, is summarized in Figure 4.

Table 4 Summary of Feedback Statements Findings (Summative Usability Study)

Table 5 Summary of Feedback Statements Findings According to Prior Experience with the RebiSmart Electromechanical Autoinjector (Summative Usability Study)

Figure 4 Percentage of participants providing the highest rating of the individual feedback statements, according to prior experience of the RebiSmart® electromechanical autoinjector (summative usability study). *Considered by RebiSmart®-experienced participants only, in comparison to RebiSmart® 2.0.

Discussion

Several studies have shown that the RebiSmart electromechanical autoinjector facilitates adherence to treatment with sc IFN β-1a among people living with MS and, in turn, may help to improve their clinical outcomes.^9–12 However, while easy to use,^3–5 the needs of people living with MS can change over time and it is therefore important that the autoinjector continues to evolve to meet such needs, as reported here.

The series of formative usability studies identified a number of recommendations to improve the existing RebiSmart device. These concerned the texture of the device to aid with handling, enlarging PIN digits to help usability, and other functionalities to assist the user (such as the ability to view injection history and settings, and audible reminders). Such improvements – including a groove to help with grip, enlarging the clicking area of the touch screen and accompanying the audible sound with an on-screen prompt – were implemented for the next-generation device, RebiSmart 3.0. A summative usability study then allowed for a full assessment of the safe use of the device by 45 participants. During this study, key use errors (and their root causes) associated with critical tasks that may lead to the potential for harm to the user were able to be identified. Examples of such errors mainly occurred with disposing of the old cartridge and inspecting the new one, and not washing hands or disinfecting the injection site. Such errors were largely thought to be due to existing personal habits of participants, which could be readily overcome by modifying future training.

Knowledge tasks were used to assess participants’ understanding of the IFU that accompanies the updated device. The root causes of many of the associated use errors arose from the participants relying on their prior understanding and habits rather than learning of special details or nuances that were detailed within the IFU. Importantly, when prompted, and on reading the IFU, the number of use errors by participants decreased. The outcomes of the knowledge tasks were that on-screen text reminders were added, and training notes were reinforced, so that the trainer would be reminded to inform the user of the common use errors.

The participants’ subjective assessment of RebiSmart 3.0 was positive, with 43 out of 45 participants considering the updated device as safe to use in its current format. Moreover, in response to the 10 user needs questions, most participants awarded the updated device the highest score (5 out of 5) to each user need question. Favorable findings were also apparent for the feedback statements that were designed to estimate ease of use and comfort with the updated device, and there was also a level of general consistency between the scores awarded by RebiSmart-experienced and RebiSmart-naïve participants. The only exception concerned robustness, which may arise from the RebiSmart-experienced users’ prior knowledge of the RebiSmart 2.0 device.

A major strength of the study is that, by assessing the RebiSmart 3.0 device through a rigorous set of critical tasks by way of simulated use scenarios and knowledge tasks, and evaluating responses to a series of user needs questions and feedback statements, the results provide a high level of confidence in the usability of the updated device. Potential limitations include the relatively small number of participants, which is typical for this type of qualitative research. An additional limitation is that the user was not in context of a real-life injection process, and the simulated test environment meant that some participants were not necessarily completing routine tasks as they typically would in their own surroundings (eg, hand washing). We also need to consider that the extended time spent in the test environment (2 hours compared with usual practice of 10 minutes) may have led participants to become fatigued by the test procedures. Further studies are also needed to evaluate the benefit of the RebiSmart 3.0 device in terms of assisting people living with MS to adhere to long-term treatment with sc IFN β-1a.

Conclusions

Our findings confirm the safety of use of the updated next-generation RebiSmart 3.0 electromechanical autoinjector, resulting from a thorough evaluation of use scenarios and knowledge tasks by the study participants and a user needs survey that demonstrated a high level of satisfaction with the device.