Collecting patient preference information using a Clinical Data Research Network: demonstrating feasibility with idiopathic pulmonary fibrosis

Introduction

Patient preference information (PPI) are data that explain the relative desirability or acceptability of attributes that vary among alternative health interventions.^1,2 The recent interest in the United States in the patient’s perspective derives from a paradigm shift by the federal government toward advancing patient engagement science and patient preference assessment.^3–5 Although PPI can be particularly useful for understanding the perspective of people with rare diseases,⁶ data collection efforts in these populations present unique challenges; notably small populations make accessing patients more difficult. One possible solution is to aggregate PPI collected from multiple sites across a research network. One such network is the PaTH Network, one of 13 Clinical Data Research Networks (CDRNs) funded by the Patient-Centered Outcomes Research Institute, a nonprofit created through the Patient Protection and Affordable Care Act. PaTH is comprised of 6 Mid-Atlantic health systems that work together as one network to learn from larger, more diverse population pools.⁷ It has a robust infrastructure for working with patients, providers and the community to understand their needs, as well as methods for patient outreach and data-sharing.⁷ Specifically, two patient partners participated weekly in the project that conceived and implemented this study. The collection of PPI was complementary to PaTH’s goals of incorporating the patient perspective into their research efforts.

Idiopathic pulmonary fibrosis (IPF) was designated as one of the disease focuses for the PaTH Network. IPF is a progressive, irreversible lung disease of unknown cause that results in lung scarring (fibrosis) and restrictive physiology.^8,9 IPF primarily affects older adults and causes symptoms, such as shortness of breath (dyspnea), coughing, limitations on physical activity,^10,11 and reduced health-related quality of life.^10,12 Complications include pulmonary hypertension, heart failure, lung cancer, and respiratory failure, which serves as the most common cause of premature death.^8,13,14 The course and severity of symptoms are highly variable, but prognosis is typically poor, with median survival 3–5 years after diagnosis.^8,15,16 IPF is rare among the general population, affecting 2–29 people per 100,00 in the United States.^17,18

The Food and Drug Administration and the European Medical Association have approved two IPF treatments that slow disease progression: nintedanib and pirfenidone.¹⁹ Based on placebo-controlled pivotal trials, the two therapies are considered to provide similar benefits^20,21 (slowing the rate of decline in lung function by 50% over 1 year^22,23 and reducing severe respiratory events)²⁴ and to have similar costs.²⁵ Both have a risk of liver toxicity.^22,23 The therapies are differentiated by their side effects impacting quality of life; nintedanib can cause diarrhea whereas pirfenidone can cause anorexia, nausea, and photosensitivity.²⁶ Nintedanib is contraindicated in patients who are on full-dose anticoagulation.²⁷ Because there have been no direct comparative trials, clinicians lack specific guidance on prescribing one or the other.²⁶ In addition to the consideration of side effects and limited contraindications, the decision to initiate therapy and the choice between the available options may heavily rely on patient preferences, an ideal circumstance to conduct a PPI study. The goal of this study was to use IPF as a proof of concept to demonstrate the feasibility of collecting PPI using a survey administered through a CDRN, in order to understand the relative acceptability of side effects that differ among alternative, similarly effective, therapies.

Material and methods

Survey development

Researchers engaged patients to understand current issues in the IPF community, including benefits and risks of treatments.²⁸ Two phases of pilot testing occurred at one PaTH site (site D). In the first phase, cognitive interviews (n=15) were used to refine the survey attributes and levels, concepts and language, as well as to evaluate which of two different preference elicitation methods (best-worst scaling or discrete choice experiment) patients preferred. The concepts, including attributes and levels, were identical across methods. The pilot survey (n=14) was administered via paper and participants completed the survey while verbalizing their responses, thoughts, and concerns about content and formatting. Researchers also asked questions as appropriate. In the second phase, participants were mailed a paper survey, completed it and returned it via mail. Unlike the previous phase, participants completed the second phase survey without assistance. The results of the pilot study informed the discrete choice experiment in this study.²⁹

Survey design

The survey consisted of three parts. The first part explained the study purpose, set expectations, and defined the attributes and levels used in the experiment. It also included two salience questions designed to slow the reader’s pace and ensure comprehension, and one warm-up task that was not part of the experimental design. The second part included the discrete choice experiment of 15 different pairs of hypothetical therapeutic profiles and an additional question comparing two profiles that were similar to the available drugs. The third section included questions about experience with either therapy.

For the second, experimental part of the survey, two hypothetical treatments were presented side-by-side and respondents selected their preferred option (Fig 1).

Figure 1 Sample discrete choice task.

The two hypothetical therapies varied according to six attributes. The attributes represented three potential benefits and three potential risks of therapy use (Table 1). Benefits represented a slowing of disease progression, not a reversal of the manifestations of the disease. The benefits included slowing of decline in lung function, slowing of worsening of shortness of breath, and slowing of worsening of persistent cough. The risks included side effects identified from therapeutic clinical trials such as gastrointestinal problems, sun sensitivity, and risk of liver toxicity. Levels for each attribute ranged from no change from current status to a significant change; levels for benefit attributes ranged from no benefit to significant slowing of disease progression and levels for risk attributes ranged from no risk to severe risk of side effect.

Table 1 Definitions of attributes and levels used

The experimental design was developed using Ngene software (Choice Metrics, Sydney, Australia). A Bayesian approach was utilized such that fixed priors were applied to indicate preference direction. The experimental design consisted of 15 choice tasks with two therapies per choice. Each attribute/level combination appeared the same number of times (10).

Recruitment and sampling strategy

Participants were adults with a confirmed diagnosis of idiopathic pulmonary fibrosis, as determined by PaTH’s participating clinicians. Sites identified an inclusive population of all patients with IPF using their electronic health records (EHRs) and then used diverse methods for outreach and recruitment: clinic intake questionnaires (some embedded in the EHR), clinical alerts, online personal health records, direct mail (surface and electronic), and social, print and broadcast media. Interested participants underwent eligibility screening for a confirmed diagnosis of IPF and consented to participate in a series of surveys being administered by PaTH. The consented cohort was 80% of the inclusive population.

The invitation to participate in the preference survey was sent to all members of the consented cohort of people with IPF. Depending on site-specific technological capabilities and individual preferences for paper versus electronic mediums, potential respondents received paper forms by mail or an email inviting them to participate that included a link to access the survey online. The survey was reviewed by the PaTH Network Protocol Review Committee under the institutional review board (IRB) for PaTH’s IPF cohort and approved by the IRB of Johns Hopkins Medicine (IRB # 60773).

Data collection procedures

Home institutions captured demographic, clinical data, and patient-reported outcomes at the point of care.³⁰ Sites used PCORnet’s Common Data Model to standardize data and facilitate its use in research across institutions.³¹ Preference data were collected via the same medium by which participants were invited to participate (paper or electronic). All data were stored in REDCap (Research Electronic Data Capture), a secure, web-based application designed to support data capture for research studies.³² Paper forms were completed at home, returned via mail using an included pre-addressed postage-paid return envelope, and research associates at each site manually entered responses into REDCap. Electronic data were collected and managed using REDCap electronic data capture tools hosted by individual sites.

Data analysis

Preference weights were estimated using conditional logistic regression with effects coding. To ease interpretability, we anchored preference weights by the “no change” category. To facilitate a complete care analysis, only data from respondents with no missing data for the 15 choice tasks were used. We calculated relative attribute importance by calculating the difference between the highest and lowest preference weights for each attribute and dividing it by the sum of all differences. We used conditional logit models to estimate preference weights. We conducted an exploratory analysis looking at the results of the one choice task that intended to simulate the choice between actual therapies on the market. All analyses were conducted using STATA version 15 (StataCorp, College Station, TX, USA).

Results

Survey development

Participant feedback from the pilot survey resulted in a number of survey changes. Participants recommended removing two of the eight original attributes because “dependence on supplemental oxygen” was irrelevant for participants that did not require oxygen and “risk of a serious adverse event” was ambiguous. Participants also raised concerns about potentially unrealistic levels and survey language, which were both altered to facilitate comprehension. Finally, the pilot work evaluated options for preference elicitation formats. Most participants accepted both best-worst scaling and discrete choice formats. Although the majority of participants found best-worst scaling easier to comprehend, the discrete choice experiment was determined to be more realistic and ultimately selected for the larger study.

Descriptive statistics

The survey was sent to 111 members of the PaTH consented cohort in January 2016. Potential participants spanned the four PaTH Network member sites. The number of potential participants varied by site ranging from 11 to 54. Overall, 33 completed surveys were returned for an overall response rate of 30%. Response rates varied by site, with sample sizes ranging from 0 to 22 (site response rates: 0–41%). The final analytic sample (n=33) was comprised of participants from three sites; site A: 6%, site B: 27%, and site C: 67% (Table 2).

Table 2 Respondent demographic and clinical characteristics

A large majority of the participants were older than 65 years of age (78%). The sample included more male (55%) and married (72%) participants and skewed toward more education, with 40% of the sample having completed a 4-year degree. All respondents had insurance coverage; the majority (85%) had public insurance including Medicare and/or Medicaid, and 15% of the sample covered by private insurance. Patient mean cough level was 10.12, mean dyspnea was 28.15, and mean fatigue score was 8.54. Medical record data identified five participants who had ever been prescribed nintedanib and 13 who had ever been prescribed pirfenidone.

Preference weights

Relative attribute importance measures the amount of choice that is attributable to a particular attribute. Slowing of decline in lung function had the greatest relative attribute importance (36%), followed by worsening of gastrointestinal problems (22%), and slowing of worsening of shortness of breath (17%). Slowing of worsening of persistent cough, risk of sun sensitivity, and risk of liver injury all accounted for <10% of attribute importance.

The conditional logit model produces preference weight estimates and standards errors (Table 3). Preference weights are the relative contribution of the attribute/level combination to the respondent’s preference for a particular alternative. These are measures of relative preference only. Therefore, it is the distance between points of an attribute that are noteworthy and the absolute value of a preference weight is not interpretable (Fig 2).

Figure 2 Preference weights for benefit/risk levels.

The difference in preference weights from no change in a benefit to the greatest amount of change in a benefit was largest for slowing of decline in lung function (5.51; [5.03, 5.99]), then for worsening of shortness of breath (2.55; [2.06, 3.03]), and least for worsening of persistent cough (0.87; [0.61, 1.14]). This can be interpreted as improving efficacy in terms of lung function was 2.16 times as important as improving efficacy in terms of shortness of breath.

Table 3 Preference weight estimates from conditional logit model with treatment choice as the dependent variable

The difference in preference weights from no side effects to the most severe side effect was largest for GI problems (−3.36; [−3.81, −2.91]). The changes for sun sensitivity (−1.31; [−1.60, −1.02]) and risk of liver injury (−1.39; [−1.76, −1.03]) were less pronounced. This can be interpreted as decreasing risk of side effects in terms of GI problems were 2.6 times as important as decreasing risk in terms of sun sensitivity or 2.4 times as important as decreasing risk in terms of liver injury.

The ratio of no slowing of decline in lung functioning and significant slowing (5.51) and the change from no GI problems to severe GI problems (3.36) indicate improving efficacy in terms of lung functioning was 1.6 times as important as decreasing risk of GI problems.

An additional choice task asked respondents to compare two profiles that resembled the available therapies on the market. The two profiles were identical in terms of all three attributes related to benefits and gastrointestinal problems. Both profiles offered moderate slowing of decline in lung function, possible slowing of worsening of shortness of breath and persistent cough, and moderate GI problems. They differed in terms of side effects; treatment A had no associated sun sensitivity, but a risk of liver injury of 1 in 1,000. Treatment B had moderate sun sensitivity but a lower risk of liver injury of 1 in 10,000. More respondents (n=19) chose treatment A than treatment B (n=14). Respondents were asked if they would actually use their choice if it were available and 16 of 19 respondents (84%) who chose treatment A said they would use the treatment and 12 of 14 respondents (86%) who chose treatment B said they would use the treatment. A comparison of respondents stated preferences in the survey to their EHR prescription history (having ever received a prescription for either medication) revealed that 13 respondents had preferences that did not match their physician’s prescribing behavior; they said they would try one of the treatments but had never received a prescription.

Discussion

This study was a proof of concept to demonstrate the feasibility of collecting PPI in a CDRN. The study pertains to IPF, a rare disease for which data collection is difficult due to its low prevalence and aggressive natural course and for which the choice between two therapeutic options is preference sensitive. We were able to successfully collect PPI data in this setting. To our knowledge, this is the first data collection effort for PPI across a research network of this kind.

Variation in site contribution to sample

There was significant variation in response across sites (ranging from 0% to 41%). Site C contributed the majority of the sample (66%), which is likely a reflection of the larger size of the eligible patient population to recruit (n=54). Although sites A and B had similar sized populations eligible for recruitment (n=24 and n=22, respectively), site A only made up 6% of the sample compared to site B contributing 27% of the sample. Site D had zero responses of 11 eligible potential respondents. Eligible participants who did not participate cited redundancy with the pilot survey (site D only) and survey length as reasons that they were not interested in participating. Other potential participants were not interested in this survey because much time had passed since they finished the rest of the surveys (site A). Disease progression and increased disease burden may have limited participants’ willingness to re-engage.

Network advantages

Results of the discrete choice experiment indicate that risk of liver injury and sun sensitivity problems were the least important attributes of therapy choice and equally important to one another, in aggregate. This may explain why the results of the final choice between two identical therapies that differed only on this attribute were similarly preferred. However, this does not explain why clinical encounters resulted in fewer prescriptions for one therapy over the other. Viewing the preference results alongside EHR data enabled by the CDRN allows for comparisons between people’s stated preferences and the decisions made in reality (prescribed medication). For instance, 13 respondents said they would try one of the treatments but had never received a prescription. This indicates there may be a mismatch in preferences, although it is not definitive given that the treatment profiles were only intended to resemble (not match exactly) the drugs available in the market. For instance, the risk of liver toxicity between actual drugs is more similar than the drug profiles suggest and the drug profiles in the study grouped GI problems together, however the actual drugs have different GI side effects: increased risk of diarrhea versus nausea/anorexia. Additional patient input is needed to determine whether specific GI effects would yield different results. Additionally, there are many other factors beyond patient preferences that inform physician prescribing behavior. For example, individual sites may develop more experience with one drug over another and develop a prescribing preference that may not reflect patient preference.

The CDRN had additional advantages. It allowed for quick identification of eligible potential participants. The consented cohort provided a readily available sample. Future preference work should target samples from networks that are already collecting data on a continuous basis. The common data model utilized by the CDRN allowed us to pool respondents and collect more data. This allowed for the aggregation of EHR and PRO data such that we did not have to repeat questions, which ultimately minimized respondent burden. This is particularly important for stated preferences studies that tend to be cognitively burdensome.

Similarly, the use of a common data collection platform (REDCap) allowed the survey to be programmed one time. Each site loaded it to their organization’s server to administer. Future multi-site preference studies should aim to streamline programming processes in a similar manner. This had the added benefit of an interface that was already familiar to the consented cohort.

Network challenges

While the CDRN offered the advantage of an accessible patient population, it also meant that the consented cohort may have been overtaxed. A challenge was to get the consented cohort to be interested in even beginning the preference survey. The preference data were different from the patient-reported outcomes data they were used to providing, which may have been confusing to participants. Future efforts to collect preference data from a CDRN might benefit from a focus on collecting preferences directly related to the other surveys being fielded among the cohort. For instance, a prioritization exercise of the outcomes that patients are reporting might be more seamless. Similarly, future data collection efforts of this type would benefit from being fielded concurrently or interspersed with other efforts to maintain respondent interest.

Limitations

There are several limitations to this analysis. First, we are limited by the small sample size. Despite combining samples across the PaTH network, the small sample limits our ability to make inferences.³³ However, this study can generate hypotheses and provide groundwork for future decision-making or patient preference work in IPF. Second, the model assumes all respondents and all tasks measure utility equally well or poorly. In reality, there may be some respondents who provide better quality responses and there may be some tasks that are easier to trade-off than others.³⁴ Third, the model does not account for unobserved systematic differences in preferences across respondents. It provides an average for the group only.³⁴ Fourth, the model results are sensitive to the levels and attributes chosen. A repeated study with different levels or definitions of levels could produce different results. For example, this study grouped many types of GI problems together (nausea/anorexia and diarrhea). Given that GI problems were the least preferred side effect, a closer look at GI side effects is needed to determine whether specific GI effects (eg, nausea/anorexia versus diarrhea) would result in different preference estimates. Finally, the results for worsening of persistent cough were not continuously increasing. The word “may” used in the level for this attribute can be interpreted with varying degrees of uncertainty and could have biased results.

Conclusion

This study is a proof of concept for the collection of PPI in a Clinical Data Research Network. It used a previously consented cohort of patients with a rare disease who would have been otherwise difficult to access or too few in number for any single site. It also allowed for the use of additional data, such as previously reported patient-reported outcomes and EHR data. Streamlined processes were favorable contributions to the study of patient preferences. Future research should continue to explore pathways to the collection and use of PPI across networks. The power of consolidated collection efforts may lead to the ability to use preference data to inform decision-making at the regional, specialty, or individual encounter level.