Patient and Caregiver Views on Measures of the Value of Health Interventions

Introduction

The use of cost-effectiveness analysis (CEA) has increased worldwide as a key input into decisions around the adoption of and reimbursement for pharmaceuticals and other health technologies. The number of published CEAs increased globally from an average of 40 per year during the period 1990–2000 to an average of 740 per year from 2010 to 2020.¹ Despite being in the forefront of CEA methods and research, the United States has lagged behind other countries in the use of CEA for decision-making.² This situation has begun to change with the advent of organizations such as the nonprofit Institute for Clinical and Economic Review as well as increasing acceptance of CEA by payers and pharmacy benefit managers.³

With this acceptance has also come a revisiting of traditional methods in CEA, which typically express health gains in terms of quality-adjusted life years (QALYs) gained, and cost-effectiveness as a ratio of incremental costs incurred to incremental QALYs gained. While recognizing the usefulness of QALYs, many researchers and policy makers have also noted that QALYs have limitations when it comes to reflecting all important domains of health.^4,5 A common criticism is that QALYs are not patient focused.⁴ Critics argue that patient preferences may not align with gains in health utilities; rather, patients are more concerned with tangible and direct impacts, like function, quality of life, and family impact.⁴ Others express concern that QALYs can be used to unfairly ration resources.⁵ Some patient advocacy groups argue that value assessments utilizing the QALY seek to cap treatments and discriminate against people with disabilities.^6,7

Researchers have begun addressing these concerns through exploration of nontraditional measures of the value to incorporate into a more comprehensive CEA, whether as an adjustment to QALY estimation or to support alternative measures of health gain.⁸ Examples include measures of equity (ie, addressing disadvantaged populations), the value of “hope” (capturing individual’s willingness to pay for a small chance of large health gains), and “real option” value (the value inherent in a small incremental gain from a treatment that allows survival long enough to achieve gains from further medical advances).⁸

While research on these domains include patient and caregiver preferences to date, there have not been efforts to assess the views of these groups on the merits of these attributes. In this exploratory and hypothesis-generating study, we sought to elicit the views of patients and caregivers on the merits of various traditional and nontraditional elements of value, as well as other domains that remain uncaptured. Using a modified Delphi approach, we aimed to solicit patient and caregiver feedback on the relative importance of traditional and nontraditional domains of value, and to determine if these views differed according to key demographic or clinical characteristics.

Methods

Overview

We used a modified Delphi technique⁹ to determine the relative importance of selected domains of value, how patients and caregivers would rank these domains, and whether attitudes might shift once we presented earlier-round results. We identified traditional and nontraditional domains of value based on those described by the “value flower” introduced by ISPOR’s Special Task Force on US Value Assessment Frameworks.⁸ We aimed to make all definitions patient-centric, given that the perspective we were most interested in was of the impact of these domains on the daily and individual burden of managing disease (or caring for someone with such a burden). Descriptions of each domain can be found in Table 1. Traditional value domains include: survival, health-related quality of life, productivity, and costs. Nontraditional value domains are comprised of: adherence-improving factors, reductions in uncertainty, fear of contagion, insurance value, severity of disease, value of hope, real option value, equity, and scientific spillovers. Our modified Delphi approach consisted of two electronically conducted survey rounds (Appendix Exhibit 1 and 2), spaced by 3 months, and a 3-week window to complete each round. The first survey round was distributed in mid-August 2021. The Tufts University Health Sciences Institutional Review Board approved survey materials, informed consent, invitation e-mails, and other supporting materials (STUDY00000799). In addition, an individual with chronic illness who also serves in several patient advocacy roles reviewed all survey materials for content and readability.

Table 1 Plain Language Definitions and Illustrative Examples of Each Domain of Value

Participants

While the existing literature on Delphi methods presents a wide range of commonly used sample sizes appropriate for conducting a Delphi study, the upper range of participants generally lies between approximately 60 and 100.^10,11 Given this range, the nature of the questions posed, and our population of interest, we aimed to recruit 75 participants, consisting of 50 patients and 25 caregivers. Participants were adults over the age of 18 who were managing a chronic medical condition or were a caregiver of a patient with a chronic medical condition. We employed a purposive, non-probability sampling approach with an aim toward recruiting a demographically diverse set of participants who also varied in terms of disease experience. We conducted initial recruitment outreach through the e-mail distribution list of an organization that has been active in discussions about value assessment (Patients Rising Now: https://patientsrisingnow.org/); for brevity, we refer to this source as a “patient activist network”. We conducted subsequent recruitment efforts through a notice in The Aging Experience newsletter (https://www.theagingexperience.com/), a weekly newsletter distributed to a diverse set of patient and caregiver subscribers that includes news and observations about aging and managing chronic disease; for brevity, we refer to this source as a “general disease advocate network”. All participants received a $50 Visa gift card only upon completion of both survey rounds.

Delphi Round 1

Round 1 utilized a structured questionnaire comprised questions about demographic information, such as age and gender, patient or caregiver status, and characterization of chronic medical conditions. Following the demographic questions, the survey asked participants to rate the thirteen domains of value on a 5-point Likert scale (0=“Not Important”, 5=“Very Important”). The survey also prompted participants to rank, in the order of most to least important, domains of value that they had previously rated as “important” or “very important”. To aid in participant understanding of the domains, the rating and ranking sections provided plain language definitions and illustrative examples for each.

At the end of the three weeks, we analyzed responses to survey round 1, excluding incomplete responses. For the rating questions, we flagged domains rated as “important” or “very important” by 55% or more of participants for retention in survey round 2. This cutoff point was determined so that retention was not based on a simple majority of favorable ratings: rather, domains would need to achieve a slightly higher standard of being rated favorably by well over half of participants. For domains of value that were flagged, we asked participants to rate and rank the domains a second time to evaluate response consistency.

We sent participants a report between survey rounds to remind them how they responded to the first round survey. We also described how a participant’s individual responses compared with responses from other participants, giving survey respondents an opportunity to reconsider their responses before completing the second survey round. Finally, we used the first round survey results to inform the development of new questions for the second survey round.

Delphi Round 2

For the second survey round, we asked participants to review and reconsider their own responses and the aggregated results from survey round 1. The second round asked the same questions as in round 1, with another round of rating and ranking the domains of value. Based on analyses of the round 1 results, we added new questions on whether to retain domains of value that did not achieve the 55% benchmark and questions about challenges patients face in taking medication, including access, out-of-pocket costs, insurance requirements, side effects, and adherence. Furthermore, we added a free-text question to gather information on additional domains of value that participants considered important to their circumstance.

Statistical Analysis

We computed the average Likert scale response for each domain of value. We used a paired t-test to determine if scores for each domain of value changed between the first and second survey rounds. We also used paired t-tests to determine if ratings differed across subgroups of interest (for example, patient vs caregiver, recruitment source, age, and gender).

To evaluate the ranked domains of value, we calculated a weighted average score for each. For each participant’s ranking, we assigned a weight of 13 to the highest ranked (most important) domain of value, a weight of 12 for next highest ranked, and so on. As above, we used a paired t-test to determine if the score for each domain of value differed between the two survey rounds.

Finally, we used multiple logistic regression analysis to identify factors associated with a respondent’s designation of a value domain as either “important” or “very important”. Covariates included participant type, recruitment source, age, gender, and survey round.

We designated results statistically significant if p < 0.05; we also reported confidence intervals for regression-based odds ratios. We conducted our analyses using Stata/SE, version 15.1.

Results

Our initial recruitment identified 79 individuals who agreed to participate. Figure 1 shows the flow of participants through the two survey stages. Of the 79 participants, 63 (79.7%) completed the first survey round and were eligible to participate in the second round. Ultimately, 58 patients and caregivers completed both rounds. Participants were more likely to be female (60.3%) and between 35 and 54 years of age (58.6%) (Table 2). About two-thirds of participants identified as patients, and we recruited most from the general disease advocate network (70.7%), rather than from the patient activist network (29.3%). Participants recruited via the patient activist network were more likely to be patients (82.4%) than caregivers, while there was a less drastic split between patients (61.0%) and caregivers (39.0%) recruited from the disease advocacy groups. Two-thirds of participants reported having or providing care for one chronic medical condition, while the remainder reported 2 or more chronic medical conditions.

Table 2 Demographic Characteristics of Sample

Figure 1 Flow of participants through the survey rounds.

At the end of survey round 1, responses for all domains of value approached our benchmark threshold, with the exception of adherence-improving factors and fear of contagion. The second survey round asked participants if they wanted to retain or “rescue” these two domains.

Survey Round 1

Using responses from all 63 participants in survey round 1, the domains of scientific spillovers, equity, and costs were rated highest, while value of hope, productivity, fear of contagion, and adherence improving factors were rated lowest. Patients rated the value of hope domain higher than caregivers (3.80 vs 3.14, p = 0.01). In comparisons of recruitment source, patient activists rated health-related quality of life (4.84 vs 3.25, p < 0.001), severity of disease (4.53 vs 3.25, p < 0.001), scientific spillovers (4.53 vs 3.59, p < 0.001), real option value (4.42 vs 3.34, p < 0.001), equity (4.37 vs 3.59, p = 0.01), survival (4.32 vs 3.50, p < 0.001), reductions in uncertainty (4.21 vs 3.30, p < 0.001), value of hope (4.05 vs 3.36, p = 0.01), and fear of contagion (3.84 vs 3.20, p = 0.03) significantly higher compared to general disease advocates.

Although survival and health-related quality of life were not among the highest rated domains individually, participants ranked these traditional domains highest when choosing between them. Additionally, cost was both rated and ranked third highest across all participants. While nontraditional value domains, such as scientific spillovers and equity, were rated highly, and patient activists were more likely to prioritize such measures compared to general disease advocates, they remained ranked below the traditional measures described above.

Survey Round 2

Approximately 60% of participants indicated a preference for “rescuing” fear of contagion and adherence improving factors, the two domains that were candidates for elimination in round 1. These preferences appeared to be driven by the beliefs of general disease advocates, who were more likely to rate these domains important than patient activists. When asked about financial barriers faced in taking medication, respondents reported out-of-pocket costs to be the biggest challenge (60.3%), while many respondents also rated both insurance requirements (53.4%) and side effects (46.6%) as important.

In response to our open-ended question about other value domains, several respondents added “quality of life” (but without any accompanying text distinguishing this from the quality of life domain already included), and one added “racial and social determinants”. Another respondent wrote that “treatment modalities that lower the patient burden, ie, remote diagnostics, telemedicine, Bluetooth connected devices, etc”. Constituted a separate and important value domain. Finally, one respondent wrote about the importance of mode of administration, describing the importance of the “variety of drug dispensaries such as liquid, pill or patch form”.

Survey Round Comparison – Ratings and Rankings

Participants could assign different ratings in the two survey rounds for any of the value domains. When comparing responses among the 58 participants who completed both survey rounds, 48% changed their rating for between 2 and 6 domains; 40% changed ratings for between 7 and 9 domains; and 9% assigned different ratings for 10 or more domains. The most frequently changed domains were reduction in uncertainty and value of hope, where 40 participants changed their rating, primarily in a negative direction. The least frequently changed ratings were for health-related quality of life and for severity of disease.

Despite the changes described above, overall ratings and rankings were consistent between survey rounds. Average ratings for each domain of value across both survey rounds are presented in Figure 2. These averages did not differ statistically between rounds for any domain. Similarly, no rankings statistically differed between survey rounds (Figure 3). Respondents ranked most highly those domains generally considered traditional measures of value, including survival, health-related quality of life, costs, and productivity.

Figure 2 Survey round 1 vs survey round 2 average rating for each domain of value in terms of importance (n=58).

Figure 3 Survey round 1 vs survey round 2 comparison of weighted average of ranked domains of value (n=58).

Factors Associated with Importance Rating

When results from both survey rounds were combined, participants recruited from the patient activist network were approximately six times more likely than general disease advocates to rate each domain of value as important or very important (Odds ratio [OR] 5.68; 95% CI 3.99, 8.07) (Table 3). Other significant factors included age, in which those aged 55–64 were nearly twice as likely as 18–34 year-olds to rate a domain as important or very important (OR 2.10; 95% CI 1.31, 3.37), and those aged 35–54 were 40% more likely to do so (OR 1.40; 95% CI 1.04, 1.89). In contrast, males were less likely to rate any domain of value as important or very important (OR 0.76; 95% CI 0.59, 0.995). Patients were also less likely than caregivers to designate any domain as important or very important (OR 0.57; 95% CI 0.43, 0.76). Findings were similar when examined separately for survey rounds 1 and 2.

Table 3 Likelihood of Rating Any Domain* (Important or Very Important) in Rounds 1 and 2

Discussion

Despite our provision of definitions and examples for all domains of value and participation in two survey rounds, participants consistently rated traditional domains of value (survival, quality of life, productivity, and costs) more highly than they rated nontraditional domains (for example, real option value, value of hope). This may be because patients and caregivers can draw on experience with traditional domains of costs, survival, productivity, and quality of life in their daily lives, but domains like value of hope or real option value are more abstract. This finding is supported by a study by Shafrin et al of advanced stage melanoma and lung cancer, which showed that patients placed more value on therapies that might confer a durable survival gain than other alternatives.¹² However, our findings contrast with those from an earlier report, which suggests that patients rate value of hope more highly than traditional outcomes, like expected survival. Lakdawalla et al found that most advanced cancer patients preferred a “hopeful gamble” to a “sure bet” therapy.¹³ The difference between our findings and those from Lakdawalla et al largely reflect differences in survey format and the sample frame, with our survey including patients and caregivers experiencing a range of chronic conditions. The survey developed by Lakdawalla et al drew on real-world examples and tradeoffs of cancer treatments, and presented participants with key features of the survival profile of therapies, such as average survival and likelihood of dying within specific timeframes. In contrast to our study and the Shafrin et al study, Lakdawalla et al also asked participants to consider their willingness to pay for therapies with differing outcomes.

The most important factor associated with respondent importance ratings was recruitment source. Respondents we recruited from the patient activist network placed more importance on all value domains than respondents recruited from the general disease advocacy group. Patient activists may believe that value frameworks limit a patient’s access to therapies and focus too much on costs.⁶ These beliefs might increase the probability that they will rank many domains of value as important. These types of differences in context and perspective are critical to the research and policy community as possible changes to standard measures of health gain are considered.

This study has a number of limitations. First, the survey’s aim was exploratory and hypothesis-generating, with limitations on the sample size and survey scope. Second, there are known limitations to the generalizability of Delphi results due to the sample size and the potential uneven distribution of preexisting opinions;¹⁴ in our case, exploration of differences across multiple matrices (eg, recruitment source and patient/caregiver status) was not possible. Third, as an exploratory exercise, the results do not establish consensus; rather, they provide important information for future research in this area. Fourth, there may have been limitations that arose from providing summary round 1 results to all participants before the administration of survey round 2 that could have skewed the results in either director or towards the sample mean. We note, however, that despite changes in individual ratings and rankings, these measures were quite consistent between rounds on an overall basis. Finally, we could not account for the level of previous knowledge each respondent may have had with regard to the nontraditional value domains. We aimed to address potential differences in background knowledge by defining the value domains and providing illustrative examples for each.

Nonetheless, we believe that this study represents the first attempt to seek patient and caregiver opinions regarding the importance of both traditional and nontraditional domains of value. Our findings indicate that patients and caregivers put significant stock in traditional and well-accepted measures of the value of health innovations, suggesting that further research into nontraditional value domains should focus on augmenting and bolstering traditional measures. In addition, our results indicate that the recruitment source of patients and caregivers may significantly impact what they consider to be important in value assessment. Further research in this area might integrate consideration of individual tradeoffs between value domains, as well as the implications of expanded value assessments in the context of fixed or constrained health system budgets.