Does It Matter How You Ask? Assessing the Impact of Failure or Effectiveness Framing on Preferences for Antibiotic Treatments in a Discrete Choice Experiment

Introduction

Discrete choice experiments (DCEs) are a commonly used method to elicit stated preferences for health treatment outcomes.^1–3 In a DCE, participants are asked to choose between different choices or alternatives, which are described using sets of pre-specified attributes and levels. The foundational theory of DCEs, the Random Utility Theory (RUT), assumes that the utility of an alternative can be derived through the compound valuation of the different attributes and levels describing the alternative.^4–6 This total utility is used by the participant to compare alternatives in a choice context and the resulting choice should represent the greatest total utility according to RUT.⁷

When developing a DCE, researchers are tasked with creating a limited set of attributes to describe the alternatives. This set of attributes should be based on a thorough qualitative process including a literature review, interviews with experts, and discussions with the (potential) consumers of the healthcare product.⁸ The goal of this process is to create a list of attributes which is thorough enough to contain those attributes which are relevant to the consumers of the healthcare product and stakeholders who will use this information without being so large as to make the study design over complex or burdensome.^9–11 A key part of this process is having attributes described in such a way that they can be clearly understood by the participants. In order to ensure that the attributes are understood, qualitative assessment of the attributes by the researchers is recommended by good practice guidelines, but often limited to checks of comprehensibility and plausibility.^9,12 Current guidelines for developing an attribute list in patient preference studies do not address the impact that framing can have on interpreting attributes.^9,13–15

The impact that framing can have on decision-making is well documented and has been studied for years in psychological fields.^16–18 This framing bias has been found to influence a participant’s understanding of the health information commonly used in DCEs, such as probability of negative outcomes or treatment risks.^19–22 Specific examples include patients preferring lung cancer treatments framed in terms of survival rather than mortality,²³ an effect so great that the framing can move an attribute from being the least important to the most important when eliciting preferences.²⁴ In one study, participants were more likely to choose a healthy lifestyle when the outcomes were framed as an increase in life expectancy vs poor health prevention.²⁵ In another study, participants were more willing to pay for colorectal screening when the tests were framed in regards to percentage of cancers found versus missed.²⁶ Most of these studies are similar in that they assessed the impact of framing for attributes that generally play a large role in decision-making (such as survival or mortality). In the field of healthcare preference assessment, only the study by Howard and Salkeld looked at the impact attribute framing had on an attribute not related to survival or mortality (ie, specificity or sensitivity of colorectal cancer screening).²⁶ None of the studies presented the different frames simultaneously as a means to possibly control for the framing effect which has previously been suggested as a method to control for framing effects.²⁷ There is thus a lack of research specific to the impact of framing in the field of patient preference assessment and the best practices to overcome it.²⁸

This article reports on a methodological study designed specifically to address this gap in knowledge by assessing the impact of framing on preferences for a hypothetical personal medical treatment. Specifically, this study aims to assess whether presenting an attribute not related to survival or mortality in a positive, negative, or combined frame impacts preference outcomes in terms of stated choices, preference estimates, importance value scores, and expected choice predictions.

Methods

Case Study and Participants

A DCE was developed to assess preferences for different antibiotic treatments when considering antibiotic resistance as an attribute of this treatment.²⁹ Antibiotics are a commonly prescribed medical treatment for bacterial infection and, like any medical treatment, they are often described in regards to treatment outcome making them an ideal subject for this preference task.^30–32 The participant group was recruited via Dynata, a commercial survey panel provider. The sample was drawn from the general Swedish population and was nationally representative in terms of age, gender, education, and geographic region. The inclusion criteria for the study were that the participant was 18–65 years of age, proficient in Swedish language, and self-reported being medically able to take antibiotics. Respondents were excluded if they could not take antibiotics (eg, due to allergies). Participants were recruited until the desired sample size (N=350 in each arm) was reached as this should provide sufficient power for the analysis.³³ Informed consent was obtained from all participants before starting and prior to completing the survey.

This study was planned in adherence to Swedish research regulations and was evaluated and approved by the Uppsala Regional Ethical Review Board (Dnr 2018/293) and complies with the Declaration of Helsinki.

Attributes, Levels and Experimental Design

The antibiotic treatments were described using five different attributes: Contribution to Antibiotic Resistance, Duration of Treatment, Likelihood of Side-Effects, Treatment Outcome, and Cost. The attributes were developed using a qualitative process involving literature reviews and four focus groups including ranking exercises with members of the general population (13 women/10 men, mean age = 38 years, age range 20–81 years). The attributes generated were then reduced and refined in discussion with research colleagues in accordance with best practice guidelines.^9,14,29 The levels for the attributes Duration of Treatment, Likelihood of Side-Effects, Treatment Outcome, and Cost were derived based on characteristics of the most commonly prescribed antibiotics used in clinical care in Sweden. Costs were included to reflect the co-pay insurance system where patients pay up to a fixed amount for medications each year which is the standard in Sweden. The levels for Contribution to Antibiotic Resistance were derived based on the report from the Swedish public health authority regarding current levels of antibiotic resistance.^34,35 The final list of attributes and levels was reviewed in eight stakeholder interviews (N=4 patients, N=1 expert on antibiotic resistance, N=1 nurse, N=2 general practitioners). All attributes were described identically for all participants except for Treatment Outcome.

In order to address the research question, three DCEs with different frames were designed for the attribute of Treatment Outcome. Treatment Outcome was framed as Failure Rate (Arm 1), Effectiveness (Arm 2), or both Effectiveness and Failure Rate combined (Arm 3). The levels for this attribute were logically identical across the three study arms, while being framed in opposite terms. For example, if in Arm 1 the Failure Rate is described as “5%”, then in Arm 2 the Effectiveness is “95%”, and in Arm 3 “95%/5%”. Thus, all treatment options were fundamentally the same even if framed in different ways. The attributes included can be found in Table 1 along with the full description presented to the participant and the levels for each attributed.

Table 1 Attributes and Levels for Each Attribute Used in the Choice Tasks Along with Explanatory Text as Presented to the Participants

For each Arm frame, the DCE design was optimized using a Bayesian D-efficient design created with Ngene 1.0 (ChoiceMetrics, 2011). The initial prior preference information was generated using best guess estimates based on literature review and expert opinions (N=1 pharmacist, N=1 nurse, N=2 general practitioners). The choice tasks consisted of a forced-choice in which a participant had to choose between two unlabeled alternatives (“Antibiotic A” or “Antibiotic B”). The decision to use a forced choice instead of an unforced choice with an opt-out alternative was based on the methodological needs of this study and desire to increase the efficiency of the design.³⁶ All participants were asked to imagine that they were planning to take antibiotics to treat a bacterial infection. The two alternatives were presented as choices the doctor gave them to treat the infection. An example choice task can be found in Figure 1.

Figure 1 Example choice task using a treatment failure framing with pop-up window.

A pilot test of the complete survey was conducted in February 2019. The pilot test consisted of 129 respondents evenly split between the different arms and recruited using the same methods and research population as the final survey. Coefficients to be used as Bayesian priors for the experimental design of the final DCE were derived from the output of multinomial logit (MNL) models fitted to the pilot data. The final Bayesian D-efficient design consisted of 48 unique choice tasks divided over 3 blocks of 16 choice tasks for each arm (respondents were randomly assigned to these blocks).

Questionnaire

The full questionnaire consisted of three different sections. The first section assessed demographic information such as age, gender, highest attained educational level, and occupation. In addition to these, two validated sets of questions were used to determine respondent’s health literacy (the Communicative and Critical Health Literacy Scale – Swedish Version; S-CCHL)³⁷ and numeracy (the 3-Item Version of the Subjective Numeracy Scale; SNS-3).³⁸ Health literacy and numeracy reflect the participants’ ability to utilize health information presented in text form or numeric form (respectively). The S-CCHL assesses health literacy using five items on a five-point Likert scale (“never (1)” to “always (5)”. The SNS-3 consists of three items on a six-point Likert scale from “not good at all/never (1)” to “extremely good/very often (6)”. Overall scores for each scale were generated by averaging their responses. Participants were then categorized as follows: scores 1/2 were classed as “inadequate”; those who had at least one score of 3 in the S-CCHL and 3/4 in the SNS-3 were classed as “problematic”; and those who consistently scored 4/5 in the S-CCHL and 5/6 in the SNS-3 were classed as “sufficient”. These scores were used to ensure the comparability of the samples in regard to their ability to comprehend written and numerical health information. Finally, six questions were used to assess experience with antibiotics, knowledge about antibiotic resistance, and self-reported health status.

The second section of the survey consisted of the DCE. Participants were instructed to imagine that they had a non-life-threatening bacterial infection and their doctor wants to prescribe antibiotics to treat the infection and avoid complications. Participants were randomized to receive a survey with Treatment Outcome framed as either treatment effectiveness, treatment failure, or a combination of both. Prior to answering the choice tasks, respondents received descriptions of bacterial infections, antibiotic treatments and antibiotic resistance. Additional information explaining the attributes and levels was available in a pop-up window which the participant could view during the task by hovering over the attribute. The attributes of Likelihood of Side-Effects and Treatment Outcome were presented as a percentage in the choice task and as a text and an icon array of these percentages in a pop-up box for further explanation to ensure participant comprehension.¹⁹ See Figure 1 for an example choice task with a pop-up window.

The final section consisted of short questions that assessed the participant’s evaluation of the questionnaire in regard to length and difficulty on a 5-point Likert scale along with an optional comments field. The questionnaire was developed using Light House Studio 9.6.1 software. The entire survey was pretested test to assess comprehensibility by eight stakeholders. This pretest resulted in two changes: rewording a knowledge related question, and the addition of an exclusion criterion regarding ability to use antibiotics.

Statistical Analysis

All analyses were conducted using STATA 14 clogit, and the mixlogit packages. A significance level of 0.05 was used for all analyses. The reference case used for all analyses was an antibiotic treatment with low contribution to antibiotic resistance, a 3-day treatment duration, 1% likelihood of side-effects, high effectiveness/low failure rate/ (95%/5%), and cost of €10.

A straight-line test was done to see if participants only chose either the left or right alternative. Potential scale parameter differences between data of the three arms were assessed using a Swait and Louviere test.³⁹ Three tests were done comparing the three model pairs (Failure-Effectiveness, Failure-Combined, Effectiveness-Combined) following the same procedure: the log likelihood of the MNL model was fitted separately in all three frames and those were tested against the log likelihood of the MNL model for the different pooled data sets, which accounted for potential scale parameter differences. The hypothesis of equal attribute-level estimates was rejected in all comparisons by means of the chi-squared test.

Parameter coefficient estimates and importance value scores for the different arms were then derived using a two-step process. First, a logit model was used to identify significant main effects, interaction effects, linearity of parameter estimates across attribute levels, and left-right bias.⁴⁰ No attributes were found to be linear across all three arms; thus, each attribute level was added to the model as a single parameter using dummy coding to allow for easy interpretation of results and willingness-to-pay estimates were not possible to calculate.⁴¹ All possible attribute level/framing-arm interaction terms were checked in an initial logit model (eg, Effectiveness* Duration: 7 days; Failure* Duration: 7 days; Combination* Duration: 7 days). Interaction terms that did not significantly add to the model were removed using backwards elimination. A mixed logit model (MIXL) was used for this analysis as it allows for the inclusion of random effects parameters accounting for naturally occurring heterogeneity of preferences in samples and results in more accurate model estimations.⁴² A MIXL model was built for each individual arm as well as the entire population in order to show the preference differences and test for interaction effects. All attributes were found to have significant heterogeneity at some level and were thus all included as random parameters. Each model was built using 14,000 Halton draws to ensure robust results.⁴³ The final MIXL model can be found in Equation 1.

In this model, the value (U) of alternative a or b in a specific decision context (c) for an individual (i) is derived as the sum of the attribute-level estimates indicating the importance value of each attribute level (), plus the specific parameter interaction term coefficients for the arm (a) of that individual (). The stochastic factors for this alternative are included in the utility function as a random error term ε. Importance value scores (IVS) and choice predictions were derived from the results of the MIXL models. The IVS were generated by standardizing the score of the attribute with the greatest absolute utility difference between the least preferred level and most preferred level to 1.⁴⁴ The attribute with the greatest absolute utility for the Failure and the Effectiveness arms was Contribution to Antibiotic Resistance and for the combination arm it was Costs.

In order to compare the differences as they may be used in an applied setting, hypothetical choice predictions were calculated as proportion of exponentiated utility attributed to an alternative from the total exponentiated utility present in a choice scenario (see Equation 2). In this equation, the uptake probability is calculated as the mean uptake of alternative (V) for an individual when asked to choose between this alternative or another (W). The choice predictions presented show the likely uptake if a participant was presented with either the most desirable antibiotic for their specific frame, the least desirable antibiotic for their specific frame, or an amoxicillin proxy. Amoxicillin was chosen as it is one of the most commonly prescribed antibiotics for outpatient usage.⁴⁵ For the purposes of this calculation, amoxicillin was described based on descriptions found in published literature. This description is having a high contribution to antibiotic resistance,⁴⁶ a treatment duration of 7 days, 10% of the users experiencing a side effect,⁴⁷ an 85%/15% effectiveness/failure rate,⁴⁸ with low costs.⁴⁹ In addition to this, individual probabilities were generated per person based on the individual parameter estimates to see what the likely individual choice predictions would be.

Results

Study Population

In total, 1124 completed responses were received. A straight-line test resulted in five responses being excluded due to selecting all right or all left alternatives; thus, the final analysis included 1119 responses. A summary of respondent demographic information can be found in Table 2. Participants generally found the survey acceptable in regards to length (M=2.72, SD=5.94) and not difficult (M= 3.42, SD=0.859) with no significant differences found between the groups (F(2)= 0.145, p=0.865) and F(2)=0.71, p = 0.932, respectively. The percentage of people who showed dominant decision-making, in which their choices were always associated with the best level of one specific attribute, did not significantly differ across arm and was 19%, 22% and 23% for the effectiveness, failure rate and the combined arm, respectively. In all arms, these lexicographical preferences were mostly registered for the resistance attribute and the cost attribute.

Table 2 Respondent Demographic Information

Preference Estimates

The preference estimates resulting from the MIXLs for each individual arm can be found in Table 3. All attributes were found to have significant parameter estimates for at least one level of each arm. Significant heterogeneity was found for at least one level of each parameter in the MIXL model in each arm indicating that the sample participants were highly varied in how they valued the different attributes when assessing antibiotics. The exception to this was the attribute of Treatment Outcome in the Effectiveness arm.

Table 3 MIXL Model Results for the Three Different Arms

In the pooled sample, no significant interaction effect was found for the attribute of Treatment Outcome despite this being the attribute which was framed differently between the arms. Significant interaction effects were found between the way that Treatment Effectiveness was framed and the attributes of Likelihood of Side-Effects and Treatment Duration. The Likelihood of Side-Effects was significantly more important to participants who saw the effectiveness framing compared to respondents who saw the Failure or combined framing. For Treatment Duration, those who saw a Failure Rate frame used this attribute to a greater extent than Effectiveness or Combined Frame. Further, they viewed a treatment lasting 7 days as somewhat better than a duration of 3 or 14 days. The preference estimates resulting from the MIXLs for all arms together with the significant framing interaction terms can be found in Table 4.

Table 4 Attribute-Level Estimates for the MIXL Model of the Pooled Results from All Arms

The different arms had comparable IVS patterns when standardizing on the most important attribute per arm. Costs and Contribution to Antibiotic Resistance were found to have the highest IVS with individual effectiveness or failure frames valuing Contribution to Antibiotic Resistance slightly higher and those who saw the combination frame valuing costs slightly higher. After these, all three arms had similar attribute valuations with Likelihood of Side-Effects, Treatment Outcome, and Treatment Duration being the third, fourth, and fifth most important attributes. The primary difference was that the failure frame had very similar IVS for Likelihood of Side-Effects and Treatment Outcome while the combination and effectiveness frame had more pronounced differences between the attribute valuations. When the IVS were standardized on the attribute of Treatment Outcome, the differences in scale parameter become more visible as the scale of the Failure framing is significantly smaller than that of Effectiveness of Combined frames. The IVS are visualized in Figure 2.

Figure 2 Attribute IVS when standardized on most important attributed (left) or on Tx outcome (right). Abbreviations: IVS, importance value scores; Tx, treatment. Notes: Key: Light grey: failure framing; striped: effectiveness framing; dark grey: combined frames.

Choice Predictions

Choice predictions based on individual parameter estimates were similar between arms. The most preferred antibiotic was predicted to be chosen by 92.3–95.2% of the participants. The least preferred antibiotic option was predicted to be chosen by 0.3–0.5% of the participants when compared to the most preferred antibiotic. The Amoxicillin was predicted to be chosen by 4.2–7.3% of the participants when compared to the most preferred antibiotic. See Table 5 for predicted choice predictions by frame.

Table 5 Choice Predictions by Arm (Individual Uptake Probability %)

Discussion

This study was the first that we know of that used a randomized design to compare the impact of framing when an attribute in a health-oriented DCE is framed as positive, negative, or a combination of different frames. This study showed that the impact of framing should not be evaluated simply in regards to the attribute in question. Rather, the way that an attribute is framed impacts the concurrent valuation of other attributes in a DCE, altering the utility of the alternative as a whole which is similar to previous findings.²⁴ These findings reflect the foundational theory underlying DCEs that the coefficients represent not only the valuation of the individual attributes but the valuation of these attributes in relation to one another and the decision context in which they are being valued.⁷

In our study, those who saw the treatment outcome framed as effectiveness were more concerned with the likelihood of side-effects in their decision-making than those who saw treatment outcome framed in terms of failure. One possible explanation for this is a negativity bias where participants’ attention is drawn away from positively framed attributes to more negative ones.^50,51 The potential presence of this bias is reflected in the IVS of the different frames. We found that participants who saw a positive frame (effectiveness) had a greater importance value for the negative attribute of likelihood of side-effects compared to those who saw a negative frame. For the negative framed participants, the importance value of these two negative attributes was essentially the same. A similar explanation could also be that participants were more averse to risk in a positive treatment context than in a negative treatment context. Thus, participants were more concerned with the risk of side-effects when they believed a treatment to be effective than when a treatment was believed to not be effective. This type of risk aversion also reflects previous psychological research.¹⁸ Interestingly, these significant differences occurred without a statistically significant difference in the valuation of the treatment effectiveness attribute itself.

This study builds upon existing research looking at how attribute framing impacts the preferences of participants when elicited in a DCE. The framing effects that we found were smaller than those found in other preference studies.^23–26 The different preference estimates did not translate into largely different uptake rate projections. These comparable choice predictions would likely not lead to drastically different decisions being made by stakeholders who use this type of information to guide decision-making. Thus, the practical impact of this framing effect may be limited.

The question then is should anything be done to account for this effect. Druckman previously argued that presenting an attribute with both frames would serve as a type of baseline, eliminating the effect of presenting only one attribute.²⁷ This was not supported in our study where the two most important attributes were reversed for the combination frame compared to the individually framed alternatives. Additionally, the combination framing did not seem to present a “middle of the road” solution as the attribute level estimates framing did not fall between the positive and negative frames for any attribute across all levels. This seems to imply that the combination frame is a new frame in and of itself rather than a way to compensate for differences in opposing frames and thus cannot be used to account for a framing effect. Howard and Salkeld proposed reviewing attributes for potential framing issues during the development of the DCE and adjusting for a framing effect in the study design by presenting both frames or randomizing to different presentations of attributes.²⁶ Our study highlights issues with these recommendations as the presentation of multiple frames resulted in a new framing bias which did not necessarily reflect a middle point between the two frames across all attributes. This was especially evident for the importance value of the two most important attributes. Further, randomization to different frames between arms does not provide the relevant information needed to account for framing biases without increasing the number of responses needed for comparison. Randomization within a survey (presenting alternating frames between choice tasks) may be a way to see how large this framing effect is, but could not account for it and would require an increase in the number of choice scenarios needed to derive the requisite parameters that the participant would essentially need to complete two DCEs.

All of these suggestions treat framing effects as something to control for. It may be that the way that an attribute is framed fundamentally changes the decision context in a way that cannot and should not be corrected for. The outcomes resulting from stated preferences in these different contexts represent the “true” preferences of the participant in that context that they are presented with. Best practice guidelines for conducting a DCE say that attributes and levels should be developed through a qualitative process including contributions from representatives of all stakeholders who participate in or use the outcomes from preference studies.^8,13,14 As this study has shown, an important aspect of this process should also be to identify attributes which may be sensitive to framing effects and see how the different frames impact their relevance to these stakeholders. These do not necessarily need to be used to correct for this effect, but rather should be used to understand the preferences in varied contexts which reflect the variance seen in consultation rooms and information presented to patients. A potential area for further research would be a qualitative study to assess why the different frames had the impact that they did. Understanding the working mechanism would give a better understanding of how the different frames impact valuation. This would also help to determine what actions, if any, should be taken in situations where a framing effect may occur. Without this qualitative information, we can only make conjectures about why different valuations were found in the different frames.

A strength of this study is that the randomization process resulted in three groups that were comparable to each other and the general Swedish population in regard to demographic and psychosocial aspects. This limits the chance that differences between the groups impacted the outcomes that were found and supports the generalizability of these findings to the general population. Further, the preference scenario used a medication which many people were already familiar with supporting the validity of the findings as participants had personal experiences to draw from. One limitation, however, is that relative lack of information on participants with a lower education levels or lower health literacy. While the demographic makeup was similar to the general Swedish population, the framing effect in these groups may be more pronounced. Another limitation is that the choice scenario was not necessarily realistic as patients are not often given different antibiotic treatment options by their treating physicians. Thus, the decision scenario was almost entirely hypothetical. Different framing effect outcomes may be found when samples consist of patients currently in treatment with decision contexts relevant to this care. One aspect that was not checked between the different arms was whether participants reinterpreted the outcomes to view them in different frames. That is, did a respondent who saw a treatment profile with 90% effectiveness see this in terms of the 90% effectiveness or as a 10% failure. While this is hypothetically possible, previous research has not shown that patients reinterpret frames in this way and we would have likely seen higher concordance between the combination arm and a single framed arm if this were the case. Future studies should consider qualitative research to see whether patients reinterpret these frames. Similarly, as this was a hypothetical situation it is not possible to know whether the participants would truly make the same choice if presented with these options in their actual care.

Conclusion

This study reaffirmed the impact that framing can have on preference outcomes. In a first of its kind design, three frames (including a combined frame) were used to evaluate the impact of framing on an attribute describing treatment effectiveness. While framing effects were found, the practical implications and interpretation of preferences outcomes was not drastically changed because of this bias. A theoretical discussion is needed to address how to move forward in light of these results. Specifically, addressing the question of whether framing effects should be controlled for, or simply better understood through improved qualitative research.