Back to Journals » Medical Devices: Evidence and Research » Volume 15

Preference Testing in Medical Devices: Current Framework and Regulatory Gaps

Authors Lewis A , Douka D, Koukoura A , Valla V , Smirthwaite A, Faarbaek SH, Vassiliadis E

Received 29 March 2022

Accepted for publication 14 June 2022

Published 6 July 2022 Volume 2022:15 Pages 199—213

DOI https://doi.org/10.2147/MDER.S368420

Checked for plagiarism Yes

Review by Single anonymous peer review

Peer reviewer comments 2

Editor who approved publication: Dr Scott Fraser



Amy Lewis,1 Despoina Douka,1 Angeliki Koukoura,1 Vasiliki Valla,1 Amie Smirthwaite,2 Susanne Holm Faarbaek,3 Efstathios Vassiliadis1

1Evnia, Copenhagen, Denmark; 2RQM+, Nottingham, England; 3BK Medical, Herlev, Denmark

Correspondence: Amy Lewis, Email [email protected]

Abstract: Preference testing is a valuable source of information that can be provided by both healthcare professionals (HCPs) and patients (users). It can be used to improve the design and development of medical devices by feeding into device usability and, ultimately, risk management. Furthermore, it can aid with selecting the most appropriate clinical endpoints to be used in the clinical evaluation of a device and increase patient engagement by incorporating patient-relevant outcomes. Preference testing is widely conducted in the food industry but is not widespread in the medical field due to limited guidelines and a lack of regulatory framework. As such, manufacturers may be unaware of the benefits of preference testing and fail to take full advantage of it, or conversely, may use inappropriate methodology and/or analyses and consequently fail to collect meaningful data. In this position paper, we aim to highlight the benefits and uses of preference testing, along with potential methods that could be used for preference testing of medical devices. A key step towards the wider implementation of preference testing in medical devices is for the publication of international standards and guidelines for the collection, assessment, and implementation of preference data into the life cycle of a medical device.

Keywords: preference testing, medical device, PPI, patient preference

Preference Testing

Preference testing refers to a research method in which a consumer/user indicates the most liked (preferred) product, usually from a choice of two. Preference testing is recommended when information on choice is required or the ordinal relationship between two products needs to be established.1 In the context of medical device design, this could be used to address preferences for different design features (eg size, shape, weight), usability factors (interaction between a device and its user(s)), or clinical endpoints (eg efficacy, side effect, etc.). This can include both HCP and patient preferences, for example, preference testing could be used to establish the clinical outcomes/benefits that a patient would like to experience (eg improved quality of life). Common methods include paired-preference testing, best-worst scaling, and discrete choice experiments2,3 (see Table 1). An overview of the advantages and disadvantages of preference testing is provided in Table 2.

Table 1 Qualitative and Quantitative Preference Methods

Table 2 Advantages and Disadvantages of Preference Testing1

Acceptance is related to preference testing and describes the overall liking or acceptability of a product and can be measured in terms of behavior, such as the purchasing of a product.4 Acceptance testing is recommended when information on the magnitude of liking/disliking of a product is required, and to establish the hedonic status (overall liking or acceptability) of two products.1 Common methods include 9-point hedonic, labelled affective magnitude and unstructured line scales (see1 for a description). Acceptance testing in this instance should not be confused with User Acceptance Testing (UAT) that may occur as part of validation and verification activities required for a medical device to conform with the relevant regulations.5 UAT is used in various research settings, for example, in clinical trials/investigations to test and verify electronic Case Report Forms (eCRFS) and the databases that store them to ensure that they are fit for purpose (ie that they store and collect data as intended).5

Although preference and acceptability are clearly related, they are not interchangeable terms as a product can be acceptable yet not preferred over an alternative product,1 thus both types of information may be tested independently. For example, a manufacturer may wish to use acceptance testing to establish if their product is liked by the intended user and to modify the product accordingly, whereas preference testing could be used to gather information on how their product compares to a competitor product (ie if their product is the preferred product). In this article, we will focus on preference testing, specifically, patient preference testing (PPI, defined below), due to recent discussions surrounding PPI within the medical field.

Preference Testing in Practice

Preference testing is used within a variety of fields, such as insurance, public policy, and product development/design. For example, preference testing is widely used within the food industry (eg in confectionery6 and meat7,8) and is often referred to as Consumer Preference Testing (CPT).2 It involves measuring consumers’ preferences of sensory properties of food (appearance, odor, texture and flavor) and can be used during product development, improvement, quality control, storage considerations, and process development.9

Preference testing has recently become a topic of discussion in the medical field, particularly within pharmaceuticals. Preference testing can be used to obtain Patient Preference Information (PPI), described by the USA Food and Drug Administration (FDA) as

qualitative or quantitative assessments of the relative desirability or acceptability to patients of specified alternatives or choices among outcomes or other attributes that differ among alternative health interventions.10

Thus, PPI describes how much one outcome/intervention is valued by a patient in comparison to alternative options, and measures the benefit-risk trade-offs that patients are willing to accept.

In the USA, the FDA has provided guidance on the use of PPI in the design and conduct of medical device clinical investigations, acknowledging the benefit that patient preferences can provide.11 For example, it is proposed that patient input can help with easier/faster recruitment in clinical investigations, with a reduction in drop-out rates and increased follow-up participation. Moreover, the data that is collected can be of greater relevance to the outcomes that matter to patients.11 The Center for Devices and Radiological Health (CDRH) has played a pivotal role in highlighting the importance of preference testing in medical devices by using PPI in their regulatory decision-making, resulting in the approval of the first device to treat obesity.12 The device (Maestro Rechargeable System) failed to meet its clinical trial endpoint of a 10% weight loss; however, a survey of patient preferences revealed patients were willing to accept the associated risks of using the device for the expected amount of weight loss the device could provide. Consequently, it was determined (based on the clinical trial, an expert panel, and patient preferences) that the benefit of the device outweighed the risks. Indeed, the use of PPI is now part of the 21st Century Cures Act (enacted in 2016). The Act requires that the FDA provides guidance on the collection of patient experience data, defined as “information about the impact of a medical condition or related therapy on a patient and the patient’s preferences for treatment”.13 The FDA must also publish a statement and its review of any patient experience data used in support of an application.13

The Regulatory Gap in Preference Testing of Medical Devices

Currently, it is uncommon for patient and/or HCP preferences to be accounted for within a medical device’s lifecycle, and even if they are, the lack of standards and regulations on how preference information should be collected, analyzed and used within decision processes are absent.14,15 Within Europe, regulation for the use of preference information within the medical field is non-existent, even in the recently enforced Medical Device Regulation (2017/745); however, there is ongoing discussion on how preference information could be used to enable regulatory decisions, mainly with respect to reimbursement and access to medical products.14–16 The PREFER project, a 5-year public-private collaborative project funded by the Innovative Medicines Initiative, aims to establish recommendations for best-practices for including PPI throughout the medical treatment lifecycle (including medicinal products, devices and services).14,16 The PREFER project highlights four key areas that need to be addressed by all involved stakeholders (eg industry, regulatory authorities, HCP, etc.) if PPI is to be used to improve medical treatments for patients:14

  1. Consensus/shared understanding of the most appropriate methodology for patient preference studies.
  2. Consensus/shared understanding of how patient preference studies can be used to inform decision processes within industry, regulatory authorities, HTA, and reimbursement agencies.
  3. Consensus/shared understanding on the best time-point to collect PPI in the product’s lifecycle.
  4. The lack of information on how to conduct patient preference studies that involve different patient populations, acute and chronic diseases, and both prevalent and uncommon diseases.

Similarly, Whichello et al15 propose that recommendations are needed for all stakeholders (eg industry, regulatory authorities, HCP, etc.) on how to collect PPI, who should best conduct patient preference studies and how results should be interpreted.15 Furthermore, reimbursement/HTA and regulatory authorities must decide the weight that is given to PPI in comparison to the information required by regulations.15

In August 2021, the PREFER project proposed a framework for the implementation of patient preferences in decision-making, which is currently under review by the European Medicines Agency (EMA) and HTA bodies (EUnetHTA).17 The aim of the framework is to address stakeholders’ concern over a lack of clear and practical framework for collecting information on patient preferences. The framework covers advice on study purpose and objectives, design and conduct, and method selection (including Discrete Choice Experiment, Swing weighting, Best-worse scaling and Threshold Technique, see Table 1), as well as how the collected PPI can be used to inform decision-making.17,18

In the United States (U.S.), the process of incorporating PPI into regulatory decision-making is a step ahead. An FDA guidance document is available on PPI and recommendations for patient preference testing (see Ref10 and (Table 3)). This guidance encourages the collection and submission of PPI for devices using the premarket approval, humanitarian device exemption (HDE), or de novo classification pathway, and emphasizes that PPI should be considered when evaluating the benefit-risk profile of a device.10 The FDA guidance also provides a list of cases in which PPI may be particularly beneficial, for instance, for devices that have a direct patient interface or devices that are intended to directly affect health-related quality of life, amongst others.10

Table 3 FDA Recommended Qualities for Patient Preference Testing

Additional guidance is available from the Patient Centered Benefit-Risk framework developed by the Medical Device Innovation Consortium (MDIC).19 This guidance focuses on the use of PPI to ensure a patient-centered approach for determining risk-benefit assessment (ie to identify risks and benefits that are most important to patients), whilst also emphasizing that PPI should be an additional source of information and should not replace clinical data on safety and performance19 (for a detailed review see20). Both these guidance documents are geared towards patient preferences and not patient and HCP preferences, as the FDA believes that usability testing is more suitable for obtaining HCP preferences; however, the FDA will also consider the preferences of care-partners and HCPs if they are relevant to the benefit-risk assessment.10

Despite the availability of guidance documents in the US, PPI data collection is not mandatory and is very much still an evolving area of activity.10 Moreover, preference testing is exempt from FDA medical device regulation in accordance with 21 CFR 812.2 (b), so long as the purpose is not for determining safety and effectiveness of the device and there is no risk to the subjects:

A device undergoing consumer preference testing, testing of a modification, or testing of a combination of two or more devices in commercial distribution, if the testing is not for the purpose of determining safety or effectiveness and does not put subjects at risk. (21 CFR Part 812.2 Investigational Device Exemptions)

For preference testing of food products, relevant international standards are available (eg ISO 13299:2016: Sensory analysis – Methodology – General guidance for establishing a sensory profile), yet standards for preference testing of medical devices is limited, with standards relating only to preference testing methodology, namely, two-sample acceptance and preference testing (ASTM E2943) and Standard Test Method for Paired Preference Test (ASTM E2263). These standards, although useful for the collection of preference information, do not provide information on how the collected data should be analyzed and used within the decision-making processes throughout a medical device’s lifecycle. This means that manufacturers may use collected preference information in any way that they see fit (provided they can provide justification for the methods used), regardless of whether the analysis they chose is the most valid and reliable method. Consequently, any preference information that is used may not accurately reflect the true preferences of the patient and/or HCP and may result in unnecessary design changes.

Preference Testing: Design, Usability, and Risks

A key implication of the lack of regulatory framework and guidance on preference testing is that the link between preference testing, usability and risk is overlooked. Preference testing, usability and risk are all inter-linked, with preference testing feeding into potential design and development changes, which may subsequently affect the usability of the device, and in turn, may present new and previously unidentified risks (see Figure 1). Preference and usability testing should not override each other, and it is of high importance that preference testing does not negatively affect usability, and consequently increases risk.

Figure 1 The potential implication of preference testing within risk management.

Preference testing should come in the early design phase, before the manufacturers start to invest sources and time in the design of the device.21 According to Pietzsch et al preference testing feeds into stage three of the five phases in the design stage-gate process of a medical device: (1) initiation – opportunity and risk analysis, (2) formulation – concept and feasibility, (3) design and development – verification and validation, (4) final validation – product launch preparation, and (5) product launch and postlaunch assessment22 (see Figure 2).

Figure 2 Stage-gate design concept for medical devices, including the conduction of preference testing. Data from sources.22,51

Usability is directly correlated with the mitigation of risks associated with correct use and use errors,23 meaning that it is an essential process in the design and development of a medical device. As preference testing is conducted prior to usability, preferences can affect usability through design changes (ie by adding/removing/adjusting features) and therefore will have an impact on risks related to device use. Any change related to device design should be followed by usability testing and implemented only if it does not pose unacceptable risks for the user. Figure 3 presents the process to follow after obtaining the results from a preference test in relation to usability. In all scenarios, usability testing includes a risk management process, by which it can be extracted whether the user’s preference affects the usability of the device or not. If a preference has no impact on usability, the proposed preference can be implemented in the development plan of the device without any further change (Scenario 1 Figure 3). If the examined preference has an unacceptable impact on usability and risk, then changes to the preference should be made (to reduce risk), or the preference should not be implemented and the next preference in the list should be examined (Scenario 2 in Figure 3). If risk management shows that preference testing results in minor but acceptable risks for the user, the preference is accepted and is implemented in the final design of the device (Scenario 3 in Figure 3). In all cases, a risk/hazard analysis must be conducted by the manufacturer to identify possible foreseen and unforeseen risks associated with the design of the device (Steps 1–4 in Figure 3).

Figure 3 Usability testing in relation to patient/physician preferences.

Potential Differences Between Patient and HCP Preferences

Although the FDA believes that usability testing is more suitable for obtaining HCP preferences than preference testing,10 preference information from HCPs can reveal differences in opinion on what the most important clinical outcomes of treatment may be.19 For example, being pain-free may seem like an obvious endpoint from both the patient and HCP, however, if patients do not consider being pain-free as a realistic endpoint, then it will not be of high preference to them.3 These potential differences in opinion between patients and HCPs are demonstrated in a recent systematic review that compared patient and HCP preferences on attributes of healthcare interventions (such as outcomes, infrastructure, safety, and so on). Findings revealed that HCPs placed more focus on attributes of structure (ie material resources, organisational structures and human resources) and care outcomes, especially mortality, compared to patients.24 Likewise, a publication regarding HCP preferences for treating patients with non-alcoholic steatohepatitis found that, although both patients and HCPs agreed on treatments that improved liver health, there were differences related to their secondary priorities. Specifically, patients focused on the impact of symptoms related to the disease, while HCPs found the impact of the progression of the disease on long-term liver damage to be as important as improving liver health. This long-term treatment was much lower in patients’ priorities.25

Further differences between patients and HCP preferences are highlighted in a recent article that established the key factors that define an orthopedic surgeons’ preferences on knee and hip implants.26 Factors related to technology were rated as the most important for choosing an implant, which included improved outcomes for patients, longevity of implant, and design and ease of implant and instrumentation, among others. Additionally, the absence of a similar implant from another vendor, and the knowledge, availability, thoroughness, and follow-up of the sales representative were also rated highly. Finance and cost were considered of lower importance.26 These findings indicate that there are preferences that can be obtained from HCPs that would be unlikely to be assessed in patients, such as the availability of similar devices, highlighting that both patient and HCP preferences provide valuable sources of information that may be missed if only one source were accounted for.

Obtaining both patient and HCP preferences is thus important to obtain a full overview of all relevant preferences that can aid with usability solutions that positively impact the risk/benefit assessment of a device. Both sources of data are imperative to create a complete device to understand and optimize design decisions. For example, during the pre-market phase, preference testing may identify sub-groups of the intended population that are willing to accept a higher risk-benefit trade-off (ie patients with higher pain or increased severity of a condition/disease may be willing to accept more risk than those with a lower pain/less severity). At the post-market stage, clinical follow-ups under the European Medical Device Regulation 2017/745 (MDR), although designed to establish performance and safety of the device, may highlight endpoints that can be used for preference testing, such as issues with usability from both patient and HCP perspectives.

Patient Preferences and Quality of Life (QoL)

PPI can result in greater transparency and accountability during product development and improve and assess the most important outcomes for patients.10,15 For example, in pharmaceuticals patient preference studies are used as part of a drug’s clinical development program to aid with selecting the most appropriate clinical endpoints (ie the endpoints that are of most value to patients) and to ensure patient-relevant outcomes are incorporated, as well as to identify the patients that may benefit most from innovative technology.27

As a result, PPI can have a direct impact on patient quality of life (QoL), defined by the World Health Organization (WHO) as

an individual’s perception of their position in life in the context of the culture and value systems in which they live and in relation to their goals, expectations, standards and concerns.28

PPI can improve QoL outcomes for patients by taking into account their goals, expectations, standards and concerns in relation to treatment outcomes. For example, a clinical trial conducted as part of the PREFER project found that patients rated their QoL as greater when treated for chronic non-malignant pain using oxycodone PR/naloxone PR, compared to previous treatment using the WHO-step I and/or WHO-step II analgesics,29 indicating the importance of patient preferences on QoL in treatment options. Likewise, QoL has been shown to have an impact on patient preferences for the treatment of multiple sclerosis.30 Patients’ treatment preferences were influenced by efficacy, side effects, and mode of administration, with some side-effects considered as too detrimental to their present QoL to be acceptable. For example, one patient reported that

It’s here and now that I’m alive, not in ten years, right? And that’s why I have to say that it’s now that my children need me and it is now that my husband must have me,30

thus highlighting that, for some patients, current QoL is more important than long-term health outcomes, and that preferences regarding the balance between treatment efficacy and current QoL differ between patients.

Taking account of PPI is thus crucial if the most appropriate treatment option is to be selected for the patient. This point has recently been highlighted by a survey examining current shared decision-making between patients and their clinicians regarding treatment options for cancer.31 The survey revealed that although 62% of patients thought that personal QoL preferences were important for treatment decision-making, only 37% reported that they discussed QoL preferences with their clinician before starting treatment.31 Thus, highlighting a demand that is not yet being met.

The importance of QoL preferences is also highlighted by the impact that QoL outcomes have on reimbursement and treatment use. For example, in England and Wales, The National Institute for Health and Care Excellence (NICE) uses an incremental cost-effectiveness ratio to determine whether a treatment is used/continued to be used, which is based upon the cost per quality-adjusted life-year (QALY).32 Similarly, in Canada, the Canadian Agency for Drugs and Technologies in Health (CADTH) recommends reimbursement for cancer drugs based upon a cost-effectiveness ratio using QALY.33 The QALY measures are based upon preference measures of QoL, thus clearly demonstrating the importance of QoL preferences on reimbursement, and ultimately influences whether a treatment is recommended/approved.33

The Value of Preference Testing in the Lifecycle of Medical Devices

Although PPI is yet to be widely implemented within medical treatments, its value within decision-making throughout the lifecycle of a medical treatment is largely acknowledged.10,14–16 For instance, a survey conducted in 2006 by the International Network of Agencies for Health Technology Assessment (INAHTA) found that involving consumers in decision-making processes “broadens the perspective of those assessments and of the advice provided to decision makers”.34 Furthermore, the use of PPI for the selection of clinical endpoints should increase participants’ willingness to be part of a clinical trial and consequently aid with clinical development, as well as improving the likelihood of the target population’s use of the medical treatment/device once marketed.35

Regarding the stage at which the incorporation of HCP/patient preferences could be most beneficial, a literature review assessing the benefits and barriers to implementation of user involvement in medical devices found that involvement of users at all stages of product development was beneficial; however, it was most beneficial if implemented during the early stages (eg design concept), design and development, prototype testing, and clinical trials.36 Moreover, a recent study conducted an in-depth investigation into the critical decision-making stages from key stakeholders (industry, regulatory authorities and reimbursement/HTA) to determine at which points PPI could be beneficial within the lifecycle of medicines. For industry stakeholders, PPI was considered valuable for decision-making processes throughout the entire lifecycle, and especially in the design and development stages.15 For the regulatory authorities, PPI was acknowledged as being important, but was not readily accepted within their decision-making stages due to a lack of recognized and structured protocols for including PPI.15 For example, one EU respondent said, “All these things are very important but you have to create a way of measuring the impact of taking into account patient preference”.15 For HTA, although mechanisms for PPI collection to help identify the most important reimbursement/HTA topics were in place and PPI was sometimes included in assessment dossiers, it was not required.15

It should be noted that there may be slight differences in opinion for medical devices at the regulatory decision-making level, due to differences in regulatory process for medicines and medical devices.15 Although the general principles surrounding the regulation of medicines and medical devices are the same (ie benefit-risk assessment and post-market monitoring), there are differences with regard to the specifics of regulation.37 For example, within medicines, pre-market assessment is conducted by the competent authorities themselves, whereas for medical devices, competent authorities work through notified bodies, with the notified bodies being responsible for ensuring compliance with the relevant regulations.37 Thus, incorporating PPI into the regulatory decision-making process in medical devices will require the involvement of both notified bodies and competent authorities.

Potential Preference Testing Methods for Medical Devices and Appropriate Method Selection

Preference testing methods can be broadly categorized into either qualitative or quantitative methods, which can involve individual subjects, groups, or a combination of both (ie concept mapping method, see Table 1). Although traditionally used within social sciences, qualitative preference testing methods can also be applied in complex medical settings (ie health services research) to assist in identifying HCPs and/or patients’ preferences, which may facilitate a stable and structured basis for medical decision-making processes.38,39 Qualitative methods provide descriptive data that is collected via participant and/or phenomenon observation and includes information that reflects the participants’ subjective experiences and decisions.40,41 Quantitative methods (also described as elicitation methods) differ from qualitative methods in terms of providing quantifiable and measurable data.40 These data can be quantified, statistically analyzed and used to inform decision-making processes.39,42 As such, both qualitative and quantitative methods can be used to collect different types of preference information from HCPs, patients and/or their legal representatives, both of which can inform regulatory and medical-related approval decisions.42

An overview of qualitative and quantitative preference testing methods that are broadly used within health care settings is presented in Table 1. Regardless of which preference testing method is chosen, an appropriate group of auditors (eg expert HCPs) should be selected, as preferences will vary based on the HCP/patient group targeted.1,43 Additional considerations should include sample size adequacy (representative sample to provide statistical confidence), study design attributes (easy to implement and comprehend) and meaningful statistics (replicable results and data analysis that will reflect “non-preference” options).3

It is also important to consider that preferences obtained using a preference test may not necessarily be an accurate reflection of preferences obtained in everyday, real-world settings. For example, when using a 9-point hedonic scale as a measure of preferences, a participant may give two products the same rating, yet still have a preference for one product over the other but have no way of indicating this44 (ie preference information is lost). As well as losing information on preferences, false preferences can also be obtained. For instance, research has indicated that paired preference testing can result in false preferences due to a response bias (eg a preconception that a preference should be given).45 For example, a study conducted to assess the preferences for two cigarettes (type A and type B) found that 80% of participants reported a preference for one of the types, despite the fact that the two types were actually identical.45 Potential methods to combat these issues are to create preference tests that do not resemble a preference task45 and to include identical “placebo” items to act as a control data that show the results of the testing conditions per se, and not actual preferences.44

Real-World Examples of Preference Testing of a Medical Device

Although preference testing is not yet widely implemented in the medical field, there are examples of its use in practice. A study on patients’ preferences was used as part of the approval decision for a novel obesity device, namely the VBLOC Maestro® Rechargeable System.42,46 The patient preference test obtained information on the maximum amount of mortality risk that a patient would accept for a percentage of total body weight loss. For example, a device with a 0.01% risk of mortality would be accepted by a patient if 10% total body weight loss for five years was the expected outcome. This information on the risk-benefit acceptance was not known prior to this preference study, and as such, the FDA did not have the clinical data to be able to quantify the trade-offs that patients were willing to accept. The preference study provided these data, which was subsequently used as part of the approval for this novel device.42,46

Likewise, a patient preference study by Medtronic on hypertension revealed that patients would accept a 20% risk of adverse events if their office-based systolic blood pressure was lowered. The study also found that a minimal acceptance benefit of less than a 2.5 mmHg reduction in office-based systolic blood pressure was needed for patients to accept the treatment risks.47 The findings differed from those observed in published literature, indicating that patients may accept a lower reduction in blood pressure (reduced benefit) and may tolerate higher risks than previously observed. These findings have important implications for devices intended to lower blood pressure, as the lowering of blood pressure by small amounts is considered beneficial to patients, despite potential treatment risks.47

Preference testing was also implemented in a study to assess patient preferences between two different types of portable infusion-pump devices.48 The preference test compared an elastomeric infusor (Baxter) with an electronically controlled mechanical pump. Despite the numerous technical benefits of the electronically controlled pumps, patients preferred the elastomeric pump (testing p-value <0.01), due to its comfort.

Finally, preference testing was used as part of a study to evaluate recommendations for the use of fluorodeoxyglucose positron emission tomography (FDG-PET) as an aid for the diagnosis of dementing neurodegenerative disorders.49 Literature searches were conducted to assess the sensitivity, specificity and accuracy (amongst others) of FDG-PET in the detection of dementing neurodegenerative disorders.49 An expert panel (the European Association of Nuclear Medicine Neuroimaging Committee and the Neuroimaging and Dementia Study Groups of the European Academy of Neurology) then voted for or against the use of FDG-PET based on the obtained evidence and their expertise. They concluded that FDG-PET is useful for all dementias and mild cognitive impairment (with the exception of amyotrophic lateral sclerosis and Huntington’s Disease) but that it was not suitable for pre-clinical conditions.49 Thus, using a preference testing method (Delphi method), a consensus was achieved regarding recommendations for the clinical use of FDG-PET. These examples thus highlight the value that PPI can add to the lifecycle of a medical device.

Challenges and Lessons Learned

Despite the benefits of preference testing, there remain numerous challenges that restrict its further development and use within the medical field. A major concern is the difficulty regarding the choice of a suitable method for conducting a preference study, due to the wide range of different methods and types of information that are required at different stages of the product lifecycle.19,35 This may result in inappropriate methods being used and thus limit the value of the data obtained. The second major concern is the current level of patient and HCP knowledge/awareness of the value of preference data in the decision-making process.50 However, within the US, guidance documents provided by the FDA are highlighting the importance of PPI and paving the way for its incorporation within medical device decision-making. Likewise, within the EU, the PREFER project is increasing awareness of the value of PPI and will be key to establishing best practices for using PPI within the medical device lifecycle.

In conclusion, preference information is a valuable source of data that can provide input into the usability, and ultimately, the benefit-risk profile of a medical device. Currently, preference testing is not covered by EU regulation and is covered by an extremely limited framework within the US Moreover, guidelines and standards for implementing preference testing with medical devices are lacking/insufficient. The result of which may lead manufacturers to employ inappropriate/unreliable preference testing methods, or simply avoid conducting preference testing altogether. This paper serves to raise awareness of the value of patient and HCP preference data and its potential use throughout a device’s lifecycle. The creation of sufficient guidelines to enable stakeholders (eg industry, regulatory, HCP, etc.) to use patient and/or HCP preference information effectively, and/or incorporation of preference testing into medical device regulations, would greatly aid with the widespread implementation of preference information in medical devices.

Disclosure

Dr Amie Smirthwaite reports on working for a medical devices consultancy. The authors report no other conflicts of interest in this work.

References

1. E2943-15, A. Standard guide for two-sample acceptance and preference testing with consumers; 2021; Available from: https://www.astm.org/e2943-15r21.html. Accessed June 16, 2022.

2. Hein K, Jaeger SR, Tom Carr B, et al. Comparison of five common acceptance and preference methods. Food Qual Prefer. 2008;19(7):651–661. doi:10.1016/j.foodqual.2008.06.001

3. FDA, Virtual ISPOR-FDA Summit 2020. Using patient preference information in medical device regulatory decisions: benefit-risk and beyond; 2020.

4. Ellis BH. Acceptance and consumer preference testing. J Dairy Sci. 1969;52(6):823–831. doi:10.3168/jds.S0022-0302(69)86658-0

5. Zbrozek A, Hebert J, Gogates G, et al. validation of electronic systems to collect patient-reported outcome (PRO) data—recommendations for clinical trial teams: report of the ISPOR ePRO systems validation good research practices task force. Value Health. 2013;16(4):480–489. doi:10.1016/j.jval.2013.04.002

6. Syam A, Ulfasari I, Ishak A, Akhmar AM. Preference test of biscuit products from pumpkin seeds (Cucurbita sp.). Enferm Clin. 2020;30(Suppl 4):375–378. doi:10.1016/j.enfcli.2019.10.115

7. Sasaki K, Ooi M, Nagura N, et al. Classification and characterization of Japanese consumers’ beef preferences by external preference mapping. J Sci Food Agric. 2017;97(10):3453–3462. doi:10.1002/jsfa.8204

8. Lawlor J, Sheehan EM, Delahunty CM, et al. Sensory characteristics and consumer preference for cooked chicken breasts from organic, corn-fed, free-range and conventionally reared animals. Int J Poult Sci. 2003;2(6):409–416. doi:10.3923/ijps.2003.409.416

9. Watts BM, Ylimaki GL, Jeffery LE, Elias LG. Basic Sensory Methods for Food Evaluation. Ottawa, ON, CA: IDRC; 1989.

10. FDA. Patient preference information - voluntary submission, review in premarket approval applications, humanitarian device exemption applications, and de novo requests, and inclusion in decision summaries and device labeling; 2016. Available from: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/patient-preference-information-voluntary-submission-review-premarket-approval-applications. Accessed June 16, 2022.

11. FDA, Patient-engagement in the design and conduct of medical device clinical investigations; 2019.

12. FDA, Patient Preference Information (PPI) in Medical Device Decision-Making; 2020.

13. Russell AL. The 21st Century Cures: Requirement on Patient Experience Data and Related Information. National Health Council; 2019.

14. de Bekker-Grob EW, Berlin C, Levitan B, et al. Giving Patients’ Preferences a Voice in Medical Treatment Life Cycle: The PREFER Public–Private Project. Springer; 2017.

15. Whichello C, Bywall KS, Mauer J, et al. An overview of critical decision-points in the medical product lifecycle: where to include patient preference information in the decision-making process? Health Policy. 2020;124(12):1325–1332. doi:10.1016/j.healthpol.2020.07.007

16. PREFER. Including the patient perspective; 2021. Available from: https://www.imi-prefer.eu/about/. Accessed June 16, 2022.

17. PREFER. CHMP & EUnetHTA parallel scientific advice: qualification of a framework and “points to consider” for method selection along with five methods for performing patient preference studies to inform regulatory and HTA body medical product decision-making; 2021.

18. PREFER, Missed the PREFER framework webinar? Recording now available; 2021.

19. Consortium, M.D.I., A framework for incorporating information on patient preferences regarding benefit and risk into regulatory assessments of new medical technology; 2015.

20. Ho M, Saha A, McCleary KK, et al. A framework for incorporating patient preferences regarding benefits and risks into regulatory assessment of medical technologies. Value Health. 2016;19(6):746–750. doi:10.1016/j.jval.2016.02.019

21. Memon M Preference tests: when should you conduct one and how? Preference Tests: when Should You Conduct One and How?; 2021. Available from: https://maze.co/blog/preference-testing/. Accessed June 16, 2022.

22. Pietzsch JB, Shluzas LA, Paté-Cornell ME, et al. Stage-gate process for the development of medical devices. J Med Device. 2009;3(2). doi:10.1115/1.3148836

23. Commission. IE, Medical devices – part 1: application of usability engineering to medical devices (IEC 62366); 2016.

24. Harrison M, Milbers K, Hudson M, et al. Do patients and health care providers have discordant preferences about which aspects of treatments matter most? Evidence from a systematic review of discrete choice experiments. BMJ Open. 2017;7(5):e014719. doi:10.1136/bmjopen-2016-014719

25. Cook N, Geier A, Schmid A, et al. Assessing physician preferences on future therapeutic options and diagnostic practices in non-alcoholic steatohepatitis. JHEP Rep. 2020;2(2):100081. doi:10.1016/j.jhepr.2020.100081

26. Burns LR, Housman M, Booth R, et al. Physician preference items: what factors matter to surgeons? Does the vendor matter? Medical Devices. 2018;11:39–49. doi:10.2147/MDER.S151647

27. Bouvy JC, Cowie L, Lovett R, et al. Use of patient preference studies in HTA decision making: a NICE perspective. Patient. 2020;13(2):145–149. doi:10.1007/s40271-019-00408-4

28. World Health Organization. The World Health Organization Quality of Life (WHOQOL); 2012. Available from: https://www.who.int/publications/i/item/WHO-HIS-HSI-Rev.2012.03. Accessed June 16, 2022.

29. van Dongen VC, Vanelderen PJL, Koopmans‐Klein G, et al. Patient preference with respect to QoL and reduction in opioid-induced constipation (OIC) after treatment with prolonged-release (PR) oxycodone/naloxone compared with previous analgesic therapy [PREFER study]. Int J Clin Pract. 2014;68(11):1364–1375. doi:10.1111/ijcp.12468

30. Lee Mortensen G, Rasmussen PV. The impact of quality of life on treatment preferences in multiple sclerosis patients. Patient Prefer Adherence. 2017;11:1789–1796. doi:10.2147/PPA.S142373

31. Williams CP, Miller‐Sonet E, Nipp RD, et al. Importance of quality‐of‐life priorities and preferences surrounding treatment decision making in patients with cancer and oncology clinicians. Cancer. 2020;126(15):3534–3541. doi:10.1002/cncr.32961

32. Rawlins MD, Culyer AJ. National institute for clinical excellence and its value judgments. BMJ. 2004;329(7459):224–227. doi:10.1136/bmj.329.7459.224

33. Raymakers AJN, Regier DA, Peacock SJ. Health-related quality of life in oncology drug reimbursement submissions in Canada: a review of submissions to the pan-Canadian oncology drug review. Cancer. 2020;126(1):148–155. doi:10.1002/cncr.32455

34. Hailey D, Nordwall M. Survey on the involvement of consumers in health technology assessment programs. Int J Technol Assess Health Care. 2006;22(4):497–499. doi:10.1017/S0266462306051427

35. Janssens R, Huys I, van Overbeeke E, et al. Opportunities and challenges for the inclusion of patient preferences in the medical product life cycle: a systematic review. BMC Med Inform Decis Mak. 2019;19(1):1–16. doi:10.1186/s12911-019-0875-z

36. Shah SGS, Robinson I. Benefits of and barriers to involving users in medical device technology development and evaluation. Int J Technol Assess Health Care. 2007;23(1):131–137. doi:10.1017/S0266462307051677

37. Parvizi N, Woods K. Regulation of medicines and medical devices: contrasts and similarities. Clin Med. 2014;14(1):6–12. doi:10.7861/clinmedicine.14-1-6

38. Pope C, Mays N. Qualitative research: reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research. BMJ. 1995;311(6996):42. doi:10.1136/bmj.311.6996.42

39. Danner M, Hummel JM, Volz F, et al. Integrating patients’ views into health technology assessment: Analytic Hierarchy Process (AHP) as a method to elicit patient preferences. Int J Technol Assess Health Care. 2011;27(4):369–375. doi:10.1017/S0266462311000523

40. Soekhai V, Whichello C, Levitan B, et al. Methods for exploring and eliciting patient preferences in the medical product lifecycle: a literature review. Drug Discov Today. 2019;24(7):1324–1331. doi:10.1016/j.drudis.2019.05.001

41. Boulkedid R, Abdoul H, Loustau M, et al. Using and reporting the delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6(6):e20476. doi:10.1371/journal.pone.0020476

42. Ho MP, Gonzalez JM, Lerner HP, et al. Incorporating patient-preference evidence into regulatory decision making. Surg Endosc. 2015;29(10):2984–2993. doi:10.1007/s00464-014-4044-2

43. E2263-12, A. Standard test method for paired preference test; 2018. Available from: www.astm.org. Accessed June 16, 2022.

44. Villegas‐ruiz X, Angulo O, O’mahony M. Hidden and false “preferences” on the structured 9‐point hedonic scale. J Sens Stud. 2008;23(6):780–790. doi:10.1111/j.1745-459X.2008.00184.x

45. Xia Y, Zhong F, O’Mahony M. Paired preference testing: false preferences and disruptive protocols. Food Sci Biotechnol. 2016;25(1):1–10. doi:10.1007/s10068-016-0001-2

46. Solutions, R.-H.H. Patient preferences considered for the first time in FDA decision to approve first-of-kind obesity device; 2015. Available from: https://www.rtihs.org/news-and-events/patient-preferences-considered-first-time-fda-decision-approve-first-kind-obesity. Accessed June 16, 2022.

47. Medtronic. Medtronic study shows patients with high blood pressure are interested in an interventional procedure treatment option; 2021.

48. Zahnd D, Aebi S, Rusterholz S, et al. A randomized crossover trial assessing patient preference for two different types of portable infusion-pump devices. Ann Oncol. 1999;10(6):727–729. doi:10.1023/A:1008334313918

49. Nobili F, Arbizu J, Bouwman F, et al. European Association of Nuclear Medicine and European Academy of Neurology recommendations for the use of brain 18 F-fluorodeoxyglucose positron emission tomography in neurodegenerative cognitive impairment and dementia: delphi consensus. Eur J Neurol. 2018;25(10):1201–1217. doi:10.1111/ene.13728

50. Janssens R, Russo S, van Overbeeke E, et al. Patient preferences in the medical product life cycle: what do stakeholders think? Semi-structured qualitative interviews in Europe and the USA. Patient-Patient-Centered Outcomes Res. 2019;12(5):513–526. doi:10.1007/s40271-019-00367-w

51. King PH, Fries RC. Design of Biomedical Devices and Systems. Second ed. Crc Press; 2008.

52. Niederberger M, Spranger J. Delphi technique in health sciences: a map. Public Health Front. 2020;8:457. doi:10.3389/fpubh.2020.00457

53. Taylor E. We agree, don’t we? The Delphi method for health environments research. HERD. 2020;13(1):11–23. doi:10.1177/1937586719887709

54. Morgan DL, Ataie J, Carder P, et al. Introducing dyadic interviews as a method for collecting qualitative data. Qual Health Res. 2013;23(9):1276–1284. doi:10.1177/1049732313501889

55. Trochim W, Kane M. Concept mapping: an introduction to structured conceptualization in health care. Int J Qual Health Care. 2005;17(3):187–191. doi:10.1093/intqhc/mzi038

56. Flynn TN, Louviere JJ, Peters TJ, et al. Best–worst scaling: what it can do for health care research and how to do it. J Health Econ. 2007;26(1):171–189. doi:10.1016/j.jhealeco.2006.04.002

57. Flynn TN. Valuing citizen and patient preferences in health: recent developments in three types of best-worst scaling. Expert Rev Pharmacoecon Outcomes Res. 2010;10(3):259–267. doi:10.1586/erp.10.29

58. Cross RM. Exploring attitudes: the case for Q methodology. Health Educ Res. 2005;20(2):206–213. doi:10.1093/her/cyg121

59. Henrikson NB, Davison BJ, Berry DL. Measuring decisional control preferences in men newly diagnosed with prostate cancer. J Psychosoc Oncol. 2011;29(6):606–618. doi:10.1080/07347332.2011.615383

60. Alex G, Wyrwich KW. Wyrwich, health utility measures and the standard gamble. Acad Emerg Med. 2003;10(4):360–363. doi:10.1111/j.1553-2712.2003.tb01349.x

61. Yen Z-S, Davis MA, Chen S-C, et al. A cost–effectiveness analysis of treatment strategies for acute uncomplicated pyelonephritis in women. Acad Emerg Med. 2003;10(4):309–314. doi:10.1111/j.1553-2712.2003.tb01341.x

62. Hauber B, Coulter J. Using the threshold technique to elicit patient preferences: an introduction to the method and an overview of existing empirical applications. Appl Health Econ Health Policy. 2020;18(1):31–46. doi:10.1007/s40258-019-00521-3

63. Devereaux PJ, Anderson DR, Gardner MJ, et al. Differences between perspectives of physicians and patients on anticoagulation in patients with atrial fibrillation: observational study. BMJ. 2001;323(7323):1218–1222. doi:10.1136/bmj.323.7323.1218

64. Tervonen T, Gelhorn H, Sri Bhashyam S, et al. MCDA swing weighting and discrete choice experiments for elicitation of patient benefit-risk preferences: a critical assessment. Pharmacoepidemiol Drug Saf. 2017;26(12):1483–1491. doi:10.1002/pds.4255

65. Sutherland HJ, Lockwood GA, Minkin S, et al. Measuring satisfaction with health care: a comparison of single with paired rating strategies. Soc Sci Med. 1989;28(1):53–58. doi:10.1016/0277-9536(89)90306-7

66. Prosser LA. Statistical methods for the analysis of discrete-choice experiments: a report of the ISPOR conjoint analysis good research practices task force. Value Health. 2016;19(4):298–299. doi:10.1016/j.jval.2016.05.002

Creative Commons License © 2022 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.